attrs() read every attribute of an object through extract_attributes_full,
so one attribute it could not read (a corrupt or unsupported attribute
message, or a heap object it could not locate) failed all of them — the
same shape as the huge-object bug, where one 8 KiB attribute hid every
attribute on a NetCDF file's root group.
- clawhdf5-format: new attribute::extract_attributes_tolerant returns the
attributes it could read plus one error per attribute it could not.
Errors in the attribute index itself (Attribute Info message, dense
heap header, B-tree) still fail, since then it is unknown which
attributes exist. extract_attributes_full is unchanged (strict); both
share one implementation.
- clawhdf5: attrs() on Group/Dataset, MmapGroup/MmapDataset and
LazyGroup/LazyDataset leaves an unreadable attribute out (documented),
and the new attrs_with_errors() returns the map with the per-attribute
errors. A value is either returned complete or not at all.
Regression test: one_unreadable_attribute_does_not_hide_the_others (h5py
writes 11 dense attributes; one message's version byte is corrupted;
before: attrs() failed with InvalidAttributeVersion(127), after: the 10
others come back with their values and one error is reported).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Group::datasets()/groups() (and the Mmap/Lazy handles) listed only hard
links, so a soft link to a dataset or group was missing, and dataset(name)
/ group(name) on a group handle could not open one. In old-style (symbol
table) groups a soft link's entry has no object header address, and the
listing failed outright trying to parse one.
The three facade handles each had their own copy of the child-listing
code; they now share group_v2::resolve_group_children, which returns hard
links plus soft links resolved to their targets (relative targets from
the group holding the link, via the new resolve_path_from). A dangling or
cyclic soft link, an external link and a user-defined link are left out —
h5py lists their names but cannot open them. Any other error met while
resolving is returned, not hidden.
Path resolution now walks a relative soft link's target from the group
holding it instead of rebuilding the path from the root (same result,
one less re-walk), and ignores "." components.
Regression test: soft_links_are_listed_as_their_targets (h5py writes
absolute, relative, group, dangling, cyclic and external links with
libver latest and earliest; listings compared with h5py for File,
MmapFile and LazyFile).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
With a 2.0 low version bound, libhdf5 stores the VDS mapping list as heap
block version 1: every entry starts with a flags byte (0x04 same file, no
file name; 0x01/0x02 file/dataset name shared with an earlier entry, whose
index is stored in place of the name). The parser treated only a leading
0x04 byte as special, so a 0x00 flags byte read as an empty (same-file)
name and shared names were read as garbage.
Decode it as H5D__virtual_load_layout does, refusing unknown flags,
forward references and block versions above 1.
Test: vds_interop::vds_mapping_block_version1_shared_names (h5py
libver=("v200","v200") with repeated long names; failed before with
"unknown dataspace selection type") plus the exact heap block as a unit
test.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
libhdf5 serializes a VDS hyperslab as version 1 (irregular, 4-byte block
corners) for the default format bounds, and as version 2 (regular, 8-byte)
for unlimited selections in the 1.10 format. Only version 3 was accepted,
so every h5py VDS written with default libver failed with "only version-3
hyperslab selections are supported" (5 libhdf5 test files in the sweep).
Decode all three versions following H5S__hyper_deserialize, including
irregular hyperslabs (a union of blocks, enumerated in row-major order as
libhdf5 iterates them) and the all-ones "unlimited" count/block marker.
SerializedSelection exposes the raw form for unlimited-mapping support.
Test: vds_interop::vds_version1_irregular_hyperslab_selections compares
default-libver h5py VDS reads (contiguous, strided and 2-D block mappings)
with libhdf5's values.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Link types 65-255 are user-defined: their target is only meaningful to
the application that registered the link class. LinkMessage::parse
rejects them with InvalidLinkType, and group traversal propagated that,
so one such link made the whole group unlistable and every path through
it unresolvable (libhdf5's tall.h5 and tudlink.h5, class 187).
Group traversal (compact and dense) now leaves user-defined links out,
the way h5py leaves out links it cannot open; reserved types (2-63) are
still an error.
Regression test: user_defined_links_do_not_break_the_listing, on
libhdf5's own tools/test/testfiles tall.h5 and tudlink.h5 (BSD-style
HDF5 licence, 10 KB and 1 KB), committed as fixtures.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
A heap ID's type is in bits 4-5 of its first byte (H5HF_ID_TYPE_MASK
0x30); bits 6-7 are the ID version. The reader took the type from bits
6-7, so every huge object ID (0x10) was decoded as a managed one and
failed — and since dense attributes are read all at once, one attribute
over the heap's 4 KiB managed limit made every attribute on its object
unreadable (netcdf4-python's issue671.nc / issue672.nc).
- Huge objects (type 1): located directly from the ID when address and
length fit in it, otherwise through the huge-object v2 B-tree (record
types 1 and 2); filtered huge objects are decoded with the heap's
pipeline and their filter mask.
- Tiny objects (type 2): read from the ID itself.
- Filtered heaps: the header's pipeline is parsed (it was skipped short,
so the header checksum was read from the wrong place), indirect-block
entries for direct blocks carry their filtered size and mask, and
direct blocks are decoded before objects are read from them.
- An unknown ID version is an error.
FractalHeapHeader gains huge_btree_address, filter_pipeline,
root_direct_block_filtered_size, root_direct_block_filter_mask,
offset_size and length_size; read_managed_object now accepts any ID type.
Regression tests (h5py-written, compared with h5py):
dense_attribute_stored_as_a_huge_heap_object, dense_group_with_a_huge_link,
dense_group_with_a_filtered_link_heap; unit tests
tiny_object_is_read_from_the_id, huge_object_with_a_direct_id,
unknown_heap_id_version_is_refused.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
A version-1 shared message reference is version, type, six reserved bytes
and then an old-style symbol table entry: link-name offset (length size),
object header address, cache type, reserved, scratch. We read the address
straight after the reserved bytes, i.e. the link-name offset, and the
committed datatype lookup failed with InvalidObjectHeaderVersion (the bytes
checked in tcompound.h5: name offset 0x10, then 0x590 = /type1). Datasets
of 1.4/1.6-era files that use a committed datatype were unreadable.
Skip the name offset. parse_shared_ref has no length size, so add
parse_shared_ref_sized and use it in every internal caller;
parse_shared_ref keeps its signature and assumes length size == offset
size. The old parse_v1_ref unit test encoded the wrong layout and now uses
the real bytes.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Compound datatype version 1 carries, per member, a dimensionality and four
dimension sizes (HDF5 before 1.4 had no array class). The parser skipped
those 28 bytes, so a member such as `f: f32[4]` came back as a single f32
at the member's offset: the compound's size was right but its members were
wrong. libhdf5 wraps such a member in an array type of the first
`dimensionality` sizes and ignores the permutation; do the same, and
reject a dimensionality above 4 as libhdf5 does.
Only files old enough to also use layout message v1 have these, so this
became reachable with the previous commit (tarrold.h5, tcompound.h5).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
HDF5 1.4/1.6-era files store the layout as version 1 or 2: version,
dimensionality, class, 5 reserved bytes, an address (contiguous and chunked
only), dimensionality 32-bit sizes (with the trailing element-size
dimension) and, for compact storage, a 32-bit size and the raw data. They
failed with InvalidLayoutVersion — 84 of the 686 files in the audit sweep,
205 datasets.
Map them onto the existing variants: chunked uses the same version-1
B-tree chunk index as version 3 and is reported as version 3, so every
chunked read path (filters, selections, caches) applies unchanged.
Contiguous size is the product of the stored dimensions, which is what
libhdf5 computes from the dataspace; a disagreement fails the reader's size
check instead of returning wrong data.
Fixtures are HDF5's own deflate.h5 (v1, chunked + deflate) and
h5ex_g_iterate.h5 (v2, contiguous, one unallocated dataset); the new
interop test compares every dataset byte for byte against h5py.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
An internal node's child pointer is an address, the child's record count
and (below the first internal level) the child subtree's total record
count. libhdf5 (H5B2__hdr_init) encodes the record count in the width of
a leaf's maximum and the subtree total in the width of cum_max_nrec for
that depth, computed level by level from the node size. The reader
guessed 2 * leaf_max and leaf_max^depth, which agree at depth 2 but not
at depth 3: a 24 000-link group's name index has depth 3, its root's
pointers were read 3 bytes wide instead of 2, and listing failed with a
garbage heap offset.
Regression tests: dense_group_with_a_three_level_name_index (h5py writes
24 000 links; listing compared with h5py) and
subtree_capacity_matches_libhdf5.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
A child indirect block in row r of a fractal heap's doubling table spans
that row's block size of heap space, so it has
log2(size) - log2(start_block_size * width) + 1 rows (libhdf5's
H5HF__dtable_size_to_rows). The reader used row - first_indirect_row + 1,
which undercounts, so every object stored past the root block's direct
rows (512 KiB with libhdf5's defaults) was unreachable: dense groups with
a few thousand long link names, or ~20 000 short ones, could not be listed.
Regression test: dense_group_whose_heap_outgrows_the_root_direct_rows
(h5py writes 2 500 links with 248-byte names; listing compared with h5py).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
CHANGELOG: upgrade notes (changed read results for max-shape files,
saturating conversions, new writer errors, format-crate API changes) and
the reader/writer correctness fixes. known-issues: the silent-wrong-data
table with before/after sweep numbers, the gaps still open, and a
correction to the Extensible Array entry, which said files we wrote were
unaffected. CLAUDE.md: clawhdf5-gpu is vector distance computation, not
I/O, and clawhdf5-filters holds only deflate backends (no Blosc).
Also a facade test that libhdf5's 20-bit N-Bit float test data reads as
libhdf5's values.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
CLAUDE.md said ZeroClaw "imports this as a Cargo feature (clawhdf5
feature flag)" and uses clawhdf5 as its memory backend; the agent crate
called itself the "ZeroClaw agent memory HDF5 backend"; the migrator
claimed to read "the ZeroClaw layout". Checked on 2026-09-25 against
ZeroClaw v0.8.5 (its latest release), the osobh/zeroclaw fork (on
v0.8.5) and both histories back to February 2026:
- no `clawhdf5` feature, dependency or memory backend has ever existed
in ZeroClaw; its backends are sqlite, lucid, postgres, qdrant,
markdown and none, behind its own `Memory` trait;
- ZeroClaw's SQLite schema is a single `memories` table (id, key,
content, category, embedding, created_at, updated_at); the
migrator's memory_chunks/sessions/entities/relations layout never
existed in ZeroClaw, so it cannot read a ZeroClaw database.
Decision: withdraw the claims (as with OpenClaw); clawhdf5 is a
standalone library with no framework integration. The migrator's
default layout is documented as its own. ZEROCLAW_VERSION keeps its name
and value (it is the persisted `edgehdf5_version` writer tag) with a
doc comment saying it is unrelated to ZeroClaw.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The docs described a "drop-in" OpenClaw memory backend enabled with
`memory.backend = "clawhdf5"`. Checked against OpenClaw's source and
docs (v2026.2.26 through v2026.9.6): that config was never valid —
v2026.2-v2026.7 accepted only "builtin"/"qmd" and rejected unknown
keys, so a Gateway given it refuses to start, and v2026.8.1 (OpenClaw
2.0) removed the key. No plugin was ever built (no manifest, no
registration, no tools), nothing was tested against OpenClaw, the
linked github.com/redclawsystems/openclaw is a 404, and
@redclaw/clawhdf5 was never published.
Decision (2026-09-25): not pursuing an OpenClaw plugin for now; ZeroClaw
is the integration target.
- Remove openclaw-integration.md, openclaw-config.md and
migration-guide.md; add docs/openclaw.md: the status, what a memory
plugin needs against v2026.9.6 (plugins.slots.memory, manifest with
kind "memory", registerMemoryCapability / MemorySearchManager,
prebuilt native packages), and what this repo has as building blocks.
- README, QUICKSTART, USE_CASES, ROADMAP (Track 7 withdrawn), CLAUDE.md
and the `openclaw` module docs describe ClawhdfBackend as what it is:
a Markdown-oriented library backend, not an OpenClaw plugin. The
QUICKSTART example is corrected (the old one called a three-argument
create that does not exist) and states its limits.
- packages/clawhdf5-node: marked unpublished and broken, "private": true
so it cannot be published by accident; its bugs (snake_case vs
camelCase fields, wrong addon path, no way to store an embedding,
wrong WAL name) are recorded in docs/known-issues.md.
- Two broken rustdoc links fixed along the way.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Makes the README's "cryptographically verifiable memory" true.
With HDF5Memory::set_signing_key(key), every checkpoint stores a signed
manifest of the store: a SHA-256 per memory record (text, embedding as
stored, channel, timestamp, session, tags, deleted flag, activation) in
a Merkle tree, plus hashes of the settings (and WAL mark), sessions and
knowledge graph. The signature, public key and manifest hashes go in
/meta; the per-record hashes in /integrity/record_hashes, so
HDF5Memory::verify(path, &public_key) can say which records changed, not
just that something did. A forged manifest fails the signature.
Decisions, as agreed:
- the key is set on the open store and never persisted;
- a signed store refuses to checkpoint without its key
(MemoryError::SigningKeyRequired); remove_signature() is the
deliberate way back to unsigned;
- checkpoints only: saves still in the WAL are not covered, and verify
reports how many there are.
The hashes cover exactly what the file persists, in the form the loader
returns it (strings lose trailing NULs; an empty WAL mark is not
written), so untouched stores verify across any number of reopen and
checkpoint cycles. MemoryError becomes #[non_exhaustive] (it already
gains variants in this unreleased version).
CLI: keygen (owner-only key file), --signing-key / CLAWHDF5_SIGNING_KEY
on writing commands (create signs immediately), verify --public-key
(JSON; exit 2 if not valid), `signed` in create/stats output.
Tests: reopen/checkpoint cycles with awkward strings (f16 and f32),
refusal without the key, wrong and rotated keys, eight kinds of edit
each detected and located, a forged manifest, unsigned stores, NULs in
text, and an edit made in place with h5py that verify pinpoints.
Cost on tank (search_harness --signing-study --full, 3 runs): ~20% of a
checkpoint (+9 ms at 10K, +89-112 ms at 100K), verify 18.6 ms / 247 ms,
32 bytes per record in the file. New deps ed25519-dalek, sha2,
rand_core: pure Rust, the no-C check passes.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
It enabled nothing — the agent layer is always built — yet the README,
QUICKSTART and USE_CASES told people to pass it. Removed, with those
snippets fixed: they now depend on the git repository (nothing is on
crates.io, so `version = "2.0"` never resolved) and USE_CASES no longer
presents the `float16` feature as half-precision storage (that is
MemoryConfig::float16, on by default for new stores).
Breaking for anyone passing `features = ["agent"]`: drop it.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
consolidation_efficiency never finished: stopped after 19 minutes on one
core while building its 100K case. Not the consolidation cycle (linear:
17 us at 100 records, 2.16 ms at 10K) but the setup — every add_memory
scores the new record's novelty against the whole working tier, the
benchmark lets that tier reach 50K, and each comparison recomputed both
norms: ~5e9 comparisons of three passes each.
ImportanceScorer::score_surprise now computes the new record's norm
once, takes each comparison in one fused, 8-lane pass (dot product and
the other norm together), and splits a working tier of 4096+ records
across threads with the `parallel` feature. Same results: tested against
the old cosine formula, including shorter, empty and zero vectors and
the parallel path. The work stays quadratic in the working-tier size by
design; with regular consolidation the tier stays near
working_capacity (100) and inserts are cheap.
The complete run takes 8 min 10 s on tank and fills in the 100K cycle
row (46.66 ms) and the memory-reduction table, which had never been
published. The binary no longer prints a record-count ratio as a
"BM25 Speedup" (never measured; Part 1 measures search latency) or
claims sub-linear cycle scaling (its own numbers grow slightly faster
than linearly).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Every undated or pre-September section re-run on one machine on one day
(tank, AMD Ryzen 7 7800X3D, 2026-09-24, commit 5c8323c), 24 commands run
serially with the load average checked before each, with the command
recorded for each section. A separate check traced every changed number
back to the raw output; its corrections are applied (e.g. the on-disk
~820 B/record is float16 plus always-deflated text on a synthetic corpus
of 40 distinct texts, not float16 alone).
Two apparent regressions were isolated rather than published:
- knowledge-graph traversal: a real bug, fixed in the previous commit;
- the write path: v2.3.0 built and run on the same machine measures the
same as today, so the old 18 us / 6.17 ms figures (undated, other
hardware) are not reproducible; float16 adds ~2 us per save and the
int8 index nothing (both isolated by switching the bench's config).
Also:
- new multimodal_bench: cross-modal search at 1K/10K records, which the
README claimed but nothing measured;
- footprint_bench reports whether it built float16 or f32 stores and
takes --f32 (it kept printing "f32" after the default changed);
- README: performance tables, the "Why" table figures and the SQLite
migration section (from the previous migrate commit);
- CHANGELOG for this branch.
Not re-run: consolidation_efficiency's 100K row and its memory-reduction
part (stopped for time), and cross_platform.sh.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
MemoryConfig::float16 now defaults to true for new stores, on
measurement: on the full LongMemEval haystack with real MiniLM
embeddings every retrieval metric matched f32 (previous commit), and at
100K the file is 48% smaller with faster checkpoints and opens.
Existing stores are unaffected: every agent store has recorded
`float16 = false` in /meta and keeps it. A test opens the v2.5.0
fixture, saves and checkpoints, and checks the embeddings are still f32
with the old rows bit-identical; another checks a new store is float16.
CLI: `create --f32` opts out; like `--f32-index` it only ever switches
the default off. `--float16` is still accepted and now a no-op.
Values beyond +-65504 are refused, so f32 remains the choice for
unnormalised vectors — the upgrade note says so.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
`longmemeval_bench --float16` builds every per-question store with
MemoryConfig::float16, so the vector stage searches half-rounded
embeddings exactly as such a store holds them.
Full longmemeval_s (500 questions, ~494 turns each) with real
all-MiniLM-L6-v2 embeddings, f32 vs float16, on tank (CUDA): identical
at every Hit@k and MRR, turn and session level, in all eight modes —
bar RRF session MRR 0.9253 vs 0.9254 and one or two flips out of ~320
in which gold session ranks first. The f32 run reproduces the published
hybrid numbers exactly. The earlier float16 evidence was synthetic
clustered data only; this is the real-embedding check.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
`HDF5Memory::search(query_embedding, query_text, &SearchOptions)` is the
store's full search path. `SearchOptions::new(k)` is plain hybrid search
with the tuned default fusion; each further stage is opt-in:
- `with_sources([..])`: only records from these source channels. The
filter applies before ranking, so a filtered search still returns up
to k results, normalised over what it can return. The HNSW pool is
over-fetched in proportion to what the filter removes, and the allowed
records are scanned exactly whenever that costs fewer distance
evaluations than the index would (~pool x M) — and as the fallback if
the pool comes back short. Keyword matches are filtered too.
- `with_rerank(ReRankConfig)` re-ranks a max(3k, 10) candidate pool by
relevance, recency, source authority and activation;
`with_confidence(ConfidenceConfig)` drops low-confidence results;
`at_time(now)` pins the recency clock.
These were reachable only through the OpenClaw backend, which is now
`search` with both on. Its Hebbian boost now goes to the k results it
returns rather than the whole 3k candidate pool. `hybrid_search` and
`hybrid_search_with` are wrappers and unchanged (tested bit for bit).
Measured on tank (search_harness --options-study --full, 3 runs): at
100K every filter — 50%, 10%, 1% of the store, and records far from the
query — returns the exact filtered top 10, and none is slower than an
unfiltered search (1%: 2.3 ms vs 4.6 ms). Re-rank + confidence costs
about 3%. A first version decided between index and exact scan by pool
size vs store size; it measured 0.976 recall at 12.3 ms on the
far-from-query filter, which is why the rule compares costs instead.
Tests: tests/search_options.rs (filter correctness and full pages via
both paths, far-from-query fallback, edge cases, equality with
hybrid_search_with, re-rank recency, confidence, boost scope).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The setting was persisted in /meta and otherwise ignored: embeddings
were always written as f32. It now does what it says.
clawhdf5-format:
- `DatasetBuilder::with_f16_data` writes IEEE binary16 (numpy float16),
rounding to nearest-even, and `make_f16_type`.
- `clawhdf5_format::float16` holds the f32 <-> f16 conversions, the one
implementation the writer, the reader and the agent all use. Checked
against the `half` crate on 16.7M f32 values and round-trips all 65536
half values; the h5py interop tests confirm the rounding matches
numpy's bit for bit (4020 values incl. ties, subnormals, overflow).
- Reading little-endian float16 as f32 has a fast path.
clawhdf5-agent:
- A float16 store writes /memory/embeddings as half precision, and
`MemoryCache::half_precision` rounds each embedding as it enters the
cache (save, update, WAL replay, and on load of a store still f32 on
disk), so memory and file agree bit for bit and a store searches the
same before and after a reopen (tested).
- Values beyond +-65504 are refused with the new
`MemoryError::InvalidEntry` rather than stored as infinity, on every
save path; batches are all or nothing, and a rejected ephemeral entry
stays in the ephemeral tier. Breaking for exhaustive matches.
- CLI: `create --float16`. Off by default.
Measured on tank, 384-dim, six runs alternating order, medians
(search_harness --float16-study --full): at 100K the file goes from
154.0 to 80.8 MiB (-48%), checkpoint 752 -> 512 ms, open 300 -> 252 ms;
vector recall@10 against an exact scan and hybrid_search latency do not
change. At 10K open is 3 ms slower. Also a test that h5py opens a whole
agent store, f32 and float16, and decodes every dataset.
Docs: README, BENCHMARKS.md ("float16 embedding storage"), CHANGELOG
(including the h5py interop fixes in the previous commit), CLAUDE.md.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
rust-version = "1.92" in [workspace.package], inherited by every crate.
1.92 is the floor: wgpu (clawhdf5-gpu) requires it, and the whole
workspace, Python bindings included, checks cleanly on it. ci-test.sh
reads the version from Cargo.toml and checks the workspace on exactly
that toolchain, so the manifests and the README badge cannot drift from
what actually builds. The badge said 1.75, below edition 2024's own
floor.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The core crates (clawhdf5, -agent, -format, -io, -filters, -ann, -accel,
-netcdf4, -cli) now build no C by default: deflate defaults to zlib-rs,
a pure-Rust port of zlib-ng, and zlib-ng becomes the opt-in
`fast-deflate`, which overrides zlib-rs wherever it is enabled. A default
build no longer needs cmake or a C compiler.
Measured on tank, both builds run alternately, three rounds, medians:
zlib-rs is within 6% of zlib-ng on every HDF5 read and write (512x512
deflate-6 chunked write 1.458 vs 1.484 ms; 64 MB compressed read 64.4
vs 65.2 ms), and compressed output is byte-identical. Details in
BENCHMARKS.md, "Deflate backend".
Getting there took two fixes the first measurement exposed:
- zlib-rs needs `std` to detect SIMD at runtime. flate2 enables it via
its default `runtime_detection`, which `default-features = false` had
switched off, leaving zlib-rs 3.5x slower on inflate. The `zlib-rs`
features now enable it.
- Both deflate paths streamed through flate2's 32 KiB read/write
wrappers. They now hand the codec the whole chunk in one call, into a
buffer sized up front (~5% on chunked writes). This also fixes a
silent short read: the streaming reader returned a truncated stream's
bytes without an error; a truncated chunk is now DecompressionError.
In clawhdf5-filters, output longer than the stated size is now an
error rather than silently cut off.
CI: ci-test.sh lints and tests the zlib-ng path, and fails if a
C-building crate (*-sys, cc, cmake) enters a core crate's default
dependency tree. The arm64 job no longer installs cmake.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
`MemoryConfig::quantized_index` now defaults to `true`. It holds a
quarter of the index memory and, with the exact re-score, is faster at
equal recall on every configuration measured: 1.63x the queries per
second on x86-64 (AVX2) and 1.18x on a Raspberry Pi 5 (NEON SDOT), with
builds 1.8x and 2.3x faster. The one argument for keeping it off — that
int8 search was slower on ARM — did not survive being measured.
Existing stores do not change. A store written with v2.6.0 or later
keeps its persisted setting. One written before the setting existed has
no stored value, and it loads as `false` rather than as the new default,
so reopening it never changes how its index is held. That case is
guarded by a real store written with the v2.5.0 CLI, committed as
`tests/fixtures/store_v2_5_0.h5` (6.8 KB): the test asserts it reopens
with an f32 index and still searches, and it fails if the load default
is changed to `true`.
The CLI needed more than a new default. `create --quantized-index`
assigned its value straight into the config, so under the new default
every CLI-created store would have been forced back to f32 unless the
caller knew to ask for int8. It is replaced by `--f32-index`, which only
ever switches the default off; `--quantized-index` is still accepted,
hidden, as a no-op, and the two conflict.
The whole agent suite passes under the new default, including the
brute-force recall oracle, now running on int8 plus re-score without
being asked to.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
On a Raspberry Pi 5 at N = 100 000 and equal recall (0.9940 vs 0.9945),
medians of three runs:
f32 33 413 ms build 6 164 QPS
int8 scalar 18 950 ms ~6 190 QPS (what v2.7.0 shipped)
int8 NEON ~17 000 ms 6 640 QPS
int8 SDOT 14 464 ms 7 267 QPS 1.18x f32, 2.3x build
The docs said quantised search stayed off by default because aarch64
"falls back to the scalar loop, where the original trade still
applies" — that it was ~13% slower than f32 there, as on x86. That was
extrapolated rather than measured, and it was wrong: x86's portable
baseline is SSE2 against hand-written AVX2 f32 kernels, but on aarch64
NEON is the baseline and the scalar loop vectorises well, so it already
matched f32. Corrected in BENCHMARKS.md, README.md and CLAUDE.md; the
released v2.7.0 changelog entry is left as it was and the correction is
recorded in a new one.
Labelled as Pi 5 figures throughout — a Pi's memory bandwidth and cache
are far below an M-series or flagship phone, so the ratios will move.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Graph degree and the build- and query-time candidate list sizes were
constants, so a deployment had no way to trade recall against memory or
query speed. They are now `MemoryConfig::hnsw_m`,
`hnsw_ef_construction` and `hnsw_ef_search`, persisted with the store
and defaulting to exactly the previous behaviour (16, 64, and a query
list that scales with `k`).
Two things the straightforward version would have got wrong:
`clawhdf5-ann` asserts a graph degree of at least 2, so a configured 0 —
from a file, or from a caller reading 0 as "use the default" — aborted
the process inside the index builder. The store clamps instead, and a
test covers it: removing the clamp makes that test panic rather than
fail.
`ef_search` and the candidate pool handed to score fusion were the same
number. Tying the pool to the new setting would mean lowering `ef` for
speed also narrows what fusion sees, quietly degrading hybrid results
through a knob that looks like it only costs time. They are now
independent.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
`read_from_disk` memory-mapped the file and then copied the entire
mapping into a `Vec` to hand to `File::from_bytes` — but `File::open`
memory-maps it itself whenever the facade's `mmap` feature is on, which
it is by default. So every open mapped the file, memcpy'd all of it, and
parsed the copy.
Store open at 100k x 384: 455 ms -> 327 ms, about 28% faster (two runs
after the change, 326.8 and 328.1 ms).
Peak memory is unchanged, which is worth saying because the opposite is
the natural assumption. The footprint harness now tracks a high-water
mark next to the retained figure, and it shows the peak falling after
the parse, during the index build — so a buffer allocated and freed
inside the parse never reaches it. Confirmed rather than assumed:
holding a deliberate extra copy of the whole file across the parse
leaves the peak exactly where it was, which is also what proved the
instrument was working before trusting its answer.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Every structure in both chunk indexes — header, index block, super
block, data block and each data block page — carries a Jenkins lookup3
checksum, and all of them were parsed past and ignored.
What that costs is not a warning but correct data. Flip one low bit of a
chunk address and the index still has the right shape, the address still
lands inside the file, and the reader returns whatever bytes now sit
there as that chunk's contents. Nothing else in the parse can tell.
Verified in both directions. The checksums accept files written by
HDF5 2.0 from 100 to 200 000 chunks — dense, sparse, gzip-filtered and
paged — which also confirms the block layouts byte for byte, since a
wrong offset would fail every file. And an interop test corrupts an
address to check the read fails instead of returning data: removing the
verification makes that test fail with "corruption produced data instead
of an error", which is what it is there to prove.
The first version of that test passed with verification disabled — it
corrupted a byte a structural check already rejected, so it proved
nothing. Worth recording, since a test that passes for the wrong reason
looks exactly like coverage.
Hand-built fixtures now stamp real checksums, as HDF5 writers do, and
the Extensible Array ones no longer describe the superseded layout.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Measured with `clawhdf5_accel::dot_i8` in place, medians of three
alternating runs at N = 100 000 x 384, same binary:
build f32 3197 ms int8 1778 ms int8+re-score 1826 ms
ef = 64 f32 13 399 QPS @ 0.9945 int8+re-score 21 848 QPS @ 0.9940
So at equal recall the quantised index is 1.63x the queries per second
and 1.8x the build speed, holding a quarter of the vectors. The earlier
"~13% of QPS" figure compared a scalar int8 loop against hand-written
AVX2 f32 kernels and was measuring the missing kernel; it is kept in
BENCHMARKS.md with that explanation rather than quietly replaced.
Still off by default, now for portability rather than performance: the
kernel is AVX2-only and aarch64 falls back to scalar, where the original
trade applies. A NEON kernel would settle it.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The Extensible Array bug reached a release because no fixture had more
chunks than fit inline, so its data blocks were never read. The Fixed
Array index had the same blind spot: nothing exercised it above a
handful of chunks, and nothing reached the paged layout at all.
Checked at 100, 5 000 and 200 000 chunks plus a sparse dataset that
leaves whole pages uninitialised. It is correct throughout — it does
keep its page-init bitmap inside the data block, which is the
difference from the Extensible Array that made assuming otherwise a
bug. Adding the tests so that stays true.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
A dataset with exactly one unlimited dimension — the ordinary
append-only case — is indexed by an Extensible Array. Only its first
few chunk entries (4 by default) sit inline in the index block, and
everything past them was read with the wrong layout. In the default
shape the 37th chunk onward came back from the wrong place: a
400-chunk dataset returned 364 wrong values while reporting success,
and beyond about a thousand chunks the read failed outright. Silently
wrong data is the worse half of that.
It survived because the only Extensible Array fixture in the suite had
three chunks — inside the inline limit — so no test ever reached a data
block.
Four layout errors, each confirmed against files written by HDF5 2.0 and
against the library source rather than inferred:
- super block `u` owns 2^(u/2) data blocks, not 2^u;
- each holds 2^((u+1)/2) * data_blk_min_elmts elements — the two
quantities double every *other* level, a half step apart;
- a super block carries a block-offset field before its data block
addresses, which was not skipped;
- the page-init bitmap belongs to the super block, one bit per page
packed across all of its data blocks and read MSB-first, rather than
living inside the data block; a paged data block also ends its prefix
with a checksum before the first page.
Where the spec left room for doubt the file settled it: decoding a
paged block's elements and reading the chunk values they address
identifies the mapping exactly, and the bitmap's 68 set bits matched
the 34 data blocks x 2 pages that 200 000 elements need, which only
holds MSB-first.
New interop tests cross every boundary — 4, 37, 400, 5 000 and 200 000
chunks, the last with paged data blocks — plus sparse (uninitialised
pages taking fill values), gzip-filtered elements and a 2-D dataset.
All three fail against the old traversal.
Writing is untouched; this was a read-path bug.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Traversal recursed one frame per level with the depth taken from the
file (a u16), and followed child addresses without asking whether they
were shared. Two crafted inputs, both reproduced before fixing:
- A node listing itself as its own child, under a header claiming 65 535
levels, overflowed the stack and aborted the process — SIGABRT, not an
error a caller can handle — from under 100 bytes.
- Levels whose children all point at one shared node below reached it
fan-out^depth times: 29.5 million records in 8 s from ~5 KB, and one
more level would exhaust memory.
Depth is now capped at 64, as the fractal heap already was; no real tree
approaches it, since even at the minimum fan-out of two that is over
2^64 records. And traversal stops once it has produced more records
than the file has bytes to hold them — a valid tree stores each record
once in its own bytes, so this bounds shared subtrees without trusting
the header's own `total_records`. Both inputs now fail in under a
millisecond.
Every B-tree v2 user goes through this collector: dense attributes, v2
groups, shared messages and chunk indexes. To show the budget never
refuses a real file, a new interop test has HDF5 2.0 write a depth-2
chunk index with 40 000 records and reads back all 160 000 values; it
fails when the budget is deliberately made too tight.
Also corrects `BM25Index::search`, which claimed to use Block-Max WAND.
It scores exhaustively, and pruning would not help the store:
`hybrid_search` needs every score because fusion normalises over them.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Every Python interop suite had stopped running on this machine: the h5py
writer round-trips, the facade suite, netCDF4 and the reference files.
`python3` is 3.14, nothing on the box has h5py, and PEP 668 refuses to
install it into a system interpreter at all — so the availability probes
all returned false and each suite skipped without failing.
A silent skip here is exactly how the v5 compound-datatype bug reached a
release, so the probes now read `CLAWHDF5_PYTHON` and `ci-test.sh` picks
up `.venv/bin/python` on its own. The detection sits at the top of the
script rather than beside the interop step, because the non-ignored
suites run in the earlier `cargo test` step and would otherwise still
miss it. `CLAWHDF5_REQUIRE_INTEROP=1` continues to turn a skip into a
failure.
Verified against a venv with h5py 3.16 / HDF5 2.0.0: 94 interop tests
across the four suites, all passing.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
`MemoryConfig::quantized_index` stores the HNSW index's own copy of the
embeddings as i8 rather than f32. At 100k x 384 that takes the index from
266 to 123 MiB and the whole reopened store from 399 to 256 MiB — 2.72x
to 1.74x the raw vectors, the largest remaining item in the footprint.
Quantised distances are approximate and `ef` cannot compensate, because
the loss is in the distances rather than in the graph: recall@10 tops out
at 0.967 against f32's 0.9995 and does not move between ef=128 and
ef=256. The store already holds the exact embeddings, though, so when the
index is quantised the query path re-scores the candidate pool against
them before fusion. That restores recall (0.9940 vs 0.9945 at ef=64) and
costs about 13% of QPS.
Off by default: it trades query speed for memory and which side is worth
more depends on the deployment. The flag is persisted in `/meta`, so a
reopened store does not silently revert to four times the index memory,
and the sidecar graph is rehydrated into the configured storage.
Also on the CLI as `create --quantized-index`.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
MemoryCache held every embedding in two places: a `Vec<Vec<f32>>` and a
flattened copy for the batched kernels, kept in lock-step on every push,
update and compaction. A store loaded from disk therefore carried the corpus
twice, plus one heap allocation per entry.
A new `cache::Embeddings` owns just the flat `[N x dim]` buffer and indexes
into it, so `embeddings[i]` still reads as a `&[f32]` row. The batch kernels
take a `VectorSet` (implemented for both `Embeddings` and `Vec<Vec<f32>>`)
instead of `&[Vec<f32>]`, so their callers and tests are unchanged. Loading no
longer unflattens what it just read.
100k 384-dim entries, reopened from disk: 505 -> 357 MiB, 3.44x -> 2.43x the
raw vectors. Recall (1.0000 at ef=64) and query latency are unchanged.
Rows are now always exactly `dim` long, shorter ones zero-padded. The old
representation allowed ragged rows, which silently misaligned the flattened
copy — every row after a wrong-length embedding — and `update` carried a
comment about falling back to a rebuild to avoid exactly that. It is now
unrepresentable. A record saved without an embedding holds a zero row and is
told apart by its norm, which is what `total_embeddings` now counts.
Measured with a counting allocator rather than RSS: freeing a structure
returns its pages to the allocator's pool, not the OS, so an RSS reading from
inside the process showed the two representations as identical.
Breaking: MemoryCache::embeddings changes type, embeddings_flat is replaced by
flat_embeddings(), rebuild_flat() is a deprecated no-op.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
reranker::rerank built its combined score from temporal decay, source
authority and Hebbian activation. RerankInput carried no relevance score, so
it could not have used one: re-ranking a candidate pool reordered it purely by
age and discarded the retriever's ordering. The OpenClaw backend re-ranks
every search, so that was its shipping behaviour.
Measured over the full LongMemEval haystack (500 questions, real MiniLM
embeddings), ordering by metadata alone costs 40.6pp of Hit@1 (11.0% vs 51.6%)
and two thirds of MRR (0.1829 vs 0.6430) — the results are the newest memories
in the pool rather than the ones answering the question.
RerankInput::relevance and ReRankConfig::relevance_weight (1.0 by default)
make relevance lead, with the metadata signals breaking near-ties. Retrieval
is preserved (Hit@1 52.0%, +0.4pp against no re-ranking; MRR -0.003) and
recency discrimination improves 6-7pp, from chance to ~52%.
A half-life sweep (1, 7, 30, 90 days) moves recency 1.4pp and MRR 0.003 —
inside the noise — because the temporal term is capped by its weight while
relevance gaps are larger. The 24-hour default is kept: there is no measured
reason to change it. The two ends of the trade-off are recorded in
BENCHMARKS.md rather than just the good news.
Breaking: RerankInput and ReRankConfig gained fields.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Bump all workspace crates, the node package and pyproject to 2.5.0, fold the
two unreleased sections together and add upgrade notes for the behaviour
changes.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The keyword stage had no stemming, so "training" and "trains" were unrelated
terms. bm25::TokenFilter::Stemmed strips common English inflections (plurals,
-ing/-ed, with consonant un-doubling) from documents and queries alike;
BM25Index::build_with and HDF5Memory::set_token_filter select it, and the
index records which filter built it so a stale one is rebuilt rather than
mixed.
Measured over the full LongMemEval haystack (500 questions, real MiniLM
embeddings) rather than adopted on principle — and it is a trade, not a win:
BM25 only Hit@1 53.8% Hit@5 75.0% Hit@10 81.6% MRR 0.6320
BM25 stemmed Hit@1 52.0% Hit@5 77.8% Hit@10 84.0% MRR 0.6320
Hybrid 0.4/0.6 Hit@1 51.6% Hit@5 81.4% Hit@10 87.8% MRR 0.6430
Hybrid stemmed Hit@1 50.2% Hit@5 81.4% Hit@10 88.2% MRR 0.6394
Conflation buys depth and costs the top rank: on BM25 alone MRR is unchanged
to four decimal places, the deeper gains exactly offsetting the rank-1 loss.
On the shipping hybrid configuration the vector stage already supplies most of
that recall, so the trade is narrower and slightly negative. Default stays
Plain; Stemmed is there for callers who want Hit@5/@10 over rank-1 precision.
The stemmer is deliberately conservative — it only strips inflections, and
only when the stem stays long enough to be meaningful, since an aggressive one
also conflates unrelated words. Tests pin both the pairs that must meet and
the pairs that must not.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Probing what QueryExpander::expand actually produces for LongMemEval questions
turned up two defects in the same helper.
`replace_word_case_insensitive` did a plain substring replace despite its
name, so acronym expansion fired inside ordinary words: "training" became
"trArtificial Intelligencening" ("ai") and "programming" became
"Pull Requestogramming" ("pr"). Nearly every acronym expansion of prose was
corrupt. Matching now requires word boundaries at both ends; real acronyms
(API, database) still expand in both directions.
The same helper searched `text.to_lowercase()` and then sliced `text` with the
offsets it found. That holds only while lowercasing preserves byte length, and
it does not — Turkish 'İ' is 2 bytes and lowercases to 3. Offsets after such a
character drifted, so output was silently corrupted ("İstanbul AI trip" lost a
character) or the slice landed inside a character or past the end and
panicked: `expand("İ AI")` was enough, from a plain query string. Matching now
walks the original string, comparing case-insensitively char by char, so
offsets are always valid.
Regression tests cover both, plus whole-word matching at string edges. The
morphological rules remain crude ("during" -> "dured"); that is a quality
limit, not a correctness bug, and is now documented as a reason to measure
before enabling expansion on a retrieval path.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
BENCHMARKS.md has recorded since the weight sweep that the 0.7/0.3 default is
strictly dominated by 0.4/0.6 over the full LongMemEval haystack, but the
shipping code never adopted it: unified_search and the OpenClaw backend both
passed 0.7/0.3. Re-running the sweep here (500 questions, real MiniLM
embeddings on a GPU) reproduces it — turn-level Hit@1 51.6% vs 44.2%, Hit@5
81.4% vs 79.2%, Hit@10 87.8% vs 85.8%, MRR 0.6430 vs 0.5856 — so both now use
hybrid::DEFAULT_FUSION, which is that operating point and carries the
reasoning. A unit test pins it.
Fusion is also selectable now. hybrid::Fusion is either Weighted { vector,
keyword } or Rrf { k }; hybrid::fuse applies either to one candidate list per
stage, and merge_vector_keyword / hybrid_search delegate to it, so the public
API is unchanged. New HDF5Memory::hybrid_search_with and
hybrid::hybrid_search_fused take a Fusion. Reciprocal rank fusion was
implemented but reachable only as a free function over a linear scan, so it
had never been compared with the weighted sum on equal terms; it is now a mode
in the LongMemEval bench (measurement to follow).
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
HDF5 1.12 revised the reference datatype (class 7) in datatype message version
4: reference types 2-4 are the new H5T_STD_REF object / dataset-region /
attribute references. Datatype::parse rejected them with
InvalidReferenceType, so any dataset of that type was unreadable.
h5py cannot write this type, which is why it had never been tested. A real
file was produced by calling the libhdf5 bundled in the h5py wheel through
ctypes (H5T_STD_REF_g, H5Rcreate_object, H5Dwrite); the 2 KB result is
committed as tests/fixtures/std_ref_hdf5_2_0.h5 with its generator,
gen_std_ref.py.
- ReferenceType gains Object2, DatasetRegion2 and Attribute, accepted only
from datatype version 4.
- read_object_references decodes Object2 elements: type(1) flags(1)
token_size(1) token, zero-padded to the element size; the token is the
target's object header address. A null reference decodes to the undefined
address; an external reference, a wrong type byte or a token that doesn't
fit is an error.
The fixture test follows both references and checks they resolve to the
objects they were created from.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Requesting a filter without chunk dimensions made the whole dataset a single
chunk. Any read, even one row, then decompresses everything, and a large
dataset cannot be decoded in parallel — which also made the new partial reads
pointless for such files.
auto_chunk_dims keeps datasets up to 1 MiB as one chunk (unchanged behaviour)
and splits larger ones by halving the dimensions in turn, so chunks keep
roughly the dataset's proportions, until a chunk is at most 1 MiB — h5py's
approach. An empty (unlimited, unwritten) dimension is treated as 1024. The
writer passes the element size through resolve_chunk_dims_for; the old
resolve_chunk_dims assumes 8-byte elements. Explicit with_chunks always wins.
Interop test: h5py reads an auto-chunked 13 MB deflate dataset, sees chunks
between 128 KiB and 1 MiB, and a small dataset still has one chunk.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Same-moment A/B on a 64 MB f64 dataset: chunked+deflate 110 -> 69 ms, chunked
72 -> 60 ms, contiguous 56 -> 30 ms.
- read_chunked_data_cached — the path the facade uses — decompressed chunks
one at a time; only the uncached reader was parallel. Cache misses are now
decoded in bounded batches (128), in parallel with the `parallel` feature.
- Every chunk was pushed into the 16 MiB chunk cache, which a larger dataset
just churns (insert, evict moments later). Chunks are cached only when the
whole dataset fits (new ChunkCache::max_bytes).
- Unfiltered chunks went file -> Vec -> aligned cache buffer -> output. They
are copied straight from the file bytes.
- The facade's typed reads convert a contiguous dataset straight from the
borrowed file bytes instead of copying it into a Vec first.
- The native little-endian fast paths allocated vec![0; n] and then overwrote
it; they now fill an uninitialised buffer in one copy (native_le_to_vec).
alloc_output requests zeroed memory from the allocator instead of reserving
and filling.
The unit test that expected unfiltered chunks to land in the decompressed
cache now asserts the new design (index reused, cache not involved).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
With libver='latest', a chunked dataset with two or more unlimited dimensions
indexes its chunks with a v2 B-tree (layout v4, index type 5). Reading one
failed with "unsupported chunked layout version=4, index_type=Some(5)".
read_btree_v2_chunks decodes record types 10 (address + scaled offsets) and 11
(address, stored size, filter mask, scaled offsets). The width of the
stored-size field is taken from the record size the tree header declares
rather than re-deriving the library's formula. Scaled offsets are multiplied
back by the chunk dimensions with overflow checks.
The chunk-index dispatch existed four times (uncached, cached, sweep and
indexed readers). The three copies outside list_chunks now call it, so every
read path — and fill-value handling and partial reads — supports every index
type from one place.
h5py interop test: plain, gzip+shuffle, a 2500-chunk tree with internal nodes,
a sparse dataset with a fill value, and a strided hyperslab.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
read_raw_data_selection computed which chunks a selection intersects, threw
the answer away, decoded the entire dataset and picked elements out of it —
for contiguous layouts too. A 64x64 window of a 64 MB deflate dataset cost
105 ms, about half a full read; every selection cost the same whatever its
size.
New partial_read module: materialise only the selection's bounding box — the
overlapping rows of a contiguous dataset (straight from the file bytes) or the
overlapping chunks (only those are decompressed) — then run the existing
extractor over that buffer with the selection translated to the box origin, so
extraction semantics are exactly the full-read ones. It declines (falling back
to the old path) for All/None, compact/virtual/storage-less layouts, and boxes
covering more than half the dataset. That window now takes 0.39 ms, one row
2.7 ms, one column 5.2 ms.
Selections are validated against the dataset shape first. They were not: a
hyperslab past an edge came back padded with zeros and a point with an
out-of-range column wrapped into the next row, returning the wrong element
with no error. Now FormatError::SelectionOutOfBounds (also rank mismatch and
overlapping blocks); the facade's fill-aware path validates too.
Tests: equivalence against a reference extraction from a full read over 60
random hyperslabs/point lists per layout (contiguous, chunked, deflate) for
ranks 1-3. New read_harness bench binary with before/after in BENCHMARKS.md.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
`parallel` joins the agent's default features, so the HNSW bulk build uses the
thread pool: cold index build at 10K records 1152 -> ~380 ms in a same-moment
A/B (the graph is identical either way). Nothing else on the measured paths
changes — ingest, checkpoint, open and steady-state query times are the same
with the feature on or off. Adds rayon to the default dependency set; opt out
with `--no-default-features --features float16,hnsw`.
Harness: `--e2e-only` runs the end-to-end section without the index
benchmarks. Note for anyone comparing numbers: this machine's absolute timings
drifted ~1.5x over a long session, so only same-moment A/B runs are
comparable.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
