`parallel` joins the agent's default features, so the HNSW bulk build uses the
thread pool: cold index build at 10K records 1152 -> ~380 ms in a same-moment
A/B (the graph is identical either way). Nothing else on the measured paths
changes — ingest, checkpoint, open and steady-state query times are the same
with the feature on or off. Adds rayon to the default dependency set; opt out
with `--no-default-features --features float16,hnsw`.
Harness: `--e2e-only` runs the end-to-end section without the index
benchmarks. Note for anyone comparing numbers: this machine's absolute timings
drifted ~1.5x over a long session, so only same-moment A/B runs are
comparable.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Profiling the build showed 90% of all distance evaluations are in back-link
pruning (40.8M of 44.9M at 10K): every overflow re-runs the diversity
heuristic pairwise over ~max_conn candidates.
The bulk build now inserts in batches: plan every node's neighbours against
the graph as it stood when the batch began (read-only, so plans are
independent), link, then prune each overflowing list once. A node gaining
several back-links in a batch is pruned once rather than once per link, so
this is faster even single-threaded (10K: 1676 -> 1074 ms). With `parallel`,
planning and pruning use rayon (10K: 388 ms; 100K: ~21 s -> 5.9 s on 16
cores). Batches start at one node and are capped at 1/16 of the linked graph
and 512 nodes; a node that raises the top layer gets a batch to itself. The
result is deterministic and identical with or without the feature (one code
path; test compares two builds byte for byte).
Parallelising within a single insert was tried first: 1.45x on 16 cores, tasks
too small. Incremental insert() stays sequential.
Recall on clustered data is unchanged or slightly better; uniform random data
dips slightly (10K, ef=64: 0.474 -> 0.444).
clawhdf5-agent's `parallel` feature now passes through to the index.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
search() collected ef candidates, then filtered out soft-deleted nodes, then
took k. When the records nearest a query had been deleted, every candidate was
a tombstone and the search returned fewer than k results — 39 of 40 queries in
the new test, which deletes each query's 40 nearest neighbours.
search_layer takes an optional skip mask: a skipped node is still pushed onto
the candidate queue (a tombstone is a valid waypoint) but never into the
result heap, so the ef result slots hold live nodes only. Build and insert
pass no mask. Recall and speed without deletions are unchanged.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Bump all workspace crates, the node package and pyproject to 2.4.0, finalize
the changelog and add upgrade notes for the search behaviour changes.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Fusion min-max normalises over every keyword match, so hybrid_search asked
BM25 for a ranked list of the whole corpus: a hash insert per posting, then a
sort of every match, then the merge sorted every candidate again to keep k.
- BM25Index::scores returns every match unsorted, accumulated in a dense array
(contributions are strictly positive, so zero means untouched). search() is
built on it with the bounded heap.
- merge_vector_keyword partitions out its top k (select_nth) and orders only
those, with the same score-then-id order.
- Both hybrid paths use scores().
Rankings are identical (equivalence tests for both changes). p50 0.24 -> 0.07
ms (1K), 2.1 -> 0.49 ms (10K), 23 -> 4.65 ms (100K).
The harness gains --fusion-study, which measured the alternative — capping the
keyword pool — and found it changes the top-10 for most queries (overlap
0.83-0.92, different #1 for 10-35%) for only a 2x saving. Not adopted.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
- Cosine distance was 1 - dot/(|a||b|), re-deriving both norms on every
evaluation in the innermost loop of build and search. The index now stores
unit vectors (prepared at build, insert, graph load and HDF5 load; the query
once per search) and uses 1 - dot. Zero vectors stay zero, giving distance 1
as before. Returned distances are unchanged.
- search_layer allocated a HashSet of visited nodes per call. It is now an
epoch-stamped u32 array in thread-local scratch, reused across calls, so
search(&self) stays shareable between threads.
- clawhdf5-accel caches the detected SIMD backend in a OnceLock.
Recall is identical. Build 2.75 -> 1.89 s (10K), ~38 -> 21 s (100K); QPS at
ef=64 22.7K -> 39K (10K), 10.4K -> 14K (100K).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
open() marked the HNSW index dirty, so the first search of every session
rebuilt it from scratch — 36 s at 100K records with the (better, slower)
heuristic build. First query after open is now 1.7 / 15 / 159 ms at
1K / 10K / 100K; what remains is the one-off keyword index build.
- clawhdf5-ann: HnswIndex::graph_to_bytes / from_graph_bytes serialize the
graph only (levels, tombstones, adjacency as u32, CRC32). The existing HDF5
serializer embeds a full copy of every vector, which would double a store
that already holds them. Loading validates everything — counts, levels vs
layer count, connection limits, every neighbour id and the layer it must
exist on — so a damaged graph, or a hostile one with a valid checksum, is an
error rather than an out-of-bounds walk during search.
- clawhdf5-agent: each checkpoint writes the graph to <store>.h5.ann (synced,
atomic, before the .h5) and records a fresh generation id in /meta. open()
loads the sidecar only if its generation matches that checkpoint; missing,
stale, damaged or mismatched sidecars are ignored and the index rebuilt.
Records appended through WAL replay join the loaded index incrementally; a
replayed Update or Tombstone invalidates it. snapshot() copies it. Only an
index that exactly mirrors the cache is saved; otherwise a stale sidecar is
removed.
- ensure_hnsw_fresh inserts records appended since the last sync instead of
rebuilding, so save_batch no longer marks the whole index dirty.
- CheckpointMeta { wal_applied, ann_generation } with *_with_meta build/write/
read functions; the *_with_mark ones delegate.
- Harness reports the one-off cold index build separately from the first query
after a reopen.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
hybrid_search rebuilt the BM25 index from scratch (re-tokenising every record)
and rewrote the whole .h5 file on every single query, so a query cost O(store
size) in both CPU and disk I/O. Steady-state p50 per the search harness:
5.5 -> 0.24 ms (1K), 49 -> 2.1 ms (10K), 884 -> 23 ms (100K).
- BM25Index is incremental: add_document / remove_document keep it exactly
equivalent to a fresh build over the same live documents (property test: 60
random op sequences compared against BM25Index::build after every step). IDF
moves to query time since it depends on the live document count. Top-k uses
a bounded heap, ties break by doc id (results were HashMap-ordered), and the
"WAND" code that computed a bound and then discarded it is removed.
- HDF5Memory keeps one index for its lifetime, built lazily. Appends are
picked up by ensure_bm25_fresh whatever path added them; delete and in-place
update report themselves; compaction drops the index. A test drives every
mutation and compares against a fresh build.
- A query no longer calls flush(). Activation boosts are marked dirty and
persisted by the next checkpoint, including a best-effort one on drop so a
search-only session keeps them (approved behaviour change). Activation
weights are capped at 16.0; they previously grew without bound.
The archived mission branch's BM25 cache was reviewed and not used: it was
invalidated by every write, so interleaved save/search still rebuilt per
query, and it changed the default fusion weights.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Neighbours were selected as the plain closest-M, for both a new node's links
and back-link pruning. On clustered data every link of a node inside a tight
cluster then goes to that same cluster, so clusters become islands a search
entering elsewhere can never reach: recall@10 was 0.87 / 0.67 / 0.31 at
1K / 10K / 100K (384-dim) and flat in ef. Uniform random data — all the
existing tests used — does not show it.
Implement the HNSW paper's Algorithm 4 with keepPrunedConnections: accept a
candidate only if it is closer to the node than to every neighbour already
accepted, then fill spare slots with the closest rejected ones. Recall@10 at
ef=64 is now 1.00 / 1.00 / 0.98 and rises with ef; uniform data improves
slightly. Build is ~3.5x slower at 10K (extra distance evaluations), to be
recovered by the distance-kernel work. The needless rayon fan-out over <=33
distances in prune_connections is gone.
Tests: a clustered-data recall test for bulk build and incremental insert
(scores 0.43 with the old selection), and a unit test of the selection rule.
Harness gains --uniform and --ann-only; before/after in BENCHMARKS.md.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
New clawhdf5-bench binary `search_harness`, the measurement baseline for the
search hot-path work. On deterministic clustered 384-dim data it reports HNSW
build time and, per ef, recall@10 against an exact scan, QPS and p50/p99; and
for HDF5Memory: ingest, checkpoint, open, first-query-after-open and
steady-state hybrid_search latency at 1K/10K (and 100K with --full). Optional
JSON output for tracking.
Baseline recorded in BENCHMARKS.md. It shows two problems: HNSW recall@10 does
not respond to ef and falls from 0.87 (1K) to 0.31 (100K) on clustered data,
and end-to-end hybrid_search is ~1000x slower than its vector stage because
every query rebuilds BM25 and rewrites the .h5 file.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Bump all workspace crates, the node package and pyproject to 2.3.0, finalize
the changelog and add upgrade notes for the behaviour changes.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
attrs() silently omitted any attribute whose datatype had no AttrValue variant
— including every Python bool, which h5py stores as an enum — plus complex,
compound and reference attributes, and cast unsigned 64-bit arrays to
I64Array so values above i64::MAX came back negative.
- numpy/h5py-style booleans (an enum of exactly FALSE=0 / TRUE=1 over an
integer base) decode as I64 / I64Array of 0/1.
- AttrValue::U64Array keeps unsigned arrays unsigned. Behaviour change: an
unsigned array attribute no longer arrives as I64Array; the netCDF-4 CF
helpers (_FillValue, valid_range) and the Python bindings handle it.
- AttrValue::Raw { datatype, shape, data } carries any other attribute
verbatim (also used when a value fails to decode as its declared type), so
the attribute list is always complete. Decodable with data_read against the
datatype; Python receives {"dtype", "shape", "data"}.
- Both new variants are writable, so attributes round-trip between files.
h5py interop tests cover reading 13 attribute kinds and h5py reading back a
compound and a u64 attribute written by clawhdf5.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
- tests/wal_properties.rs — deterministic generator, reproducible by seed:
everything appended is read back intact (300 cases), and after ANY damage
to the file (bit flips, truncation, inserted/deleted bytes, duplicated or
rotated regions, overwritten ranges; 1500 cases) reading never panics and
yields an exact prefix of what was written — the guarantee the chained CRC
exists to give. Opening for append then repairs the tail and a new entry
lands right behind the surviving prefix.
- tests/crash_recovery.rs — builds the on-disk images a process crash can
leave and reopens each against a model of what was acknowledged: an image
after every operation (random saves, in-place updates, checkpoints, small
wal_max_entries), the checkpoint window (new .h5 + not-yet-truncated WAL)
over several rounds, and the WAL torn at every byte length, which must
recover the checkpoint plus a prefix of the operations logged since.
- fuzz/fuzz_wal_replay — arbitrary bytes as a WAL: read and open-for-append
must not panic, and open() must not change what is replayable. Based on the
target from the clawmates mission branch (4aee2fa), with the repair
property added. The CI fuzz step now covers both fuzz crates.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
The VDS resolver joined the source file name stored in the HDF5 file straight
onto the opened file's directory. That name is untrusted: an absolute path
replaces the base directory outright and `..` components climb out of it, so
a crafted file could make the reader open any path the process can reach.
Only plain relative paths of normal components are accepted now; anything
else resolves to "source not found".
Co-Authored-By: Claude Fable 5.1 <[email protected]>
- Path resolution follows soft links in both old-style (symbol table, cache
type 2) and new-style (compact and dense Link message) groups: absolute and
relative targets, links to groups, links through links, with a depth limit
so a link cycle is NestingDepthExceeded rather than a hang. A dangling link
reports the target it could not find. Previously every soft link was
PathNotFound.
- An external link is FormatError::ExternalLinkUnsupported { filename,
object_path } instead of a misleading PathNotFound.
- Message 0x0007 (External Data Files) is now a known MessageType, and a
dataset carrying it is FormatError::ExternalDataFilesUnsupported. Such a
dataset has no data address in this file, so it would otherwise be read as
"never written" and answered with fill values — wrong data, no error.
- Dense link iteration is shared between hard-link listing and the new
symbolic-link lookup; entry listing behaviour is unchanged.
- h5py interop test for both libver settings.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
HDF5 allocates lazily: a chunk nobody wrote doesn't exist in the file, and a
dataset nobody wrote has no data address. Such regions must read as the
dataset's fill value. There was no Fill Value message parser at all, so:
- a sparse chunked dataset read its holes as zeros — silently wrong whenever
the fill value isn't zero (h5py `fillvalue=-1` came back as 0);
- a dataset that was created but never written failed with NoDataAllocated /
"no address for chunked layout" where h5py returns a filled array.
New clawhdf5_format::fill_value: parses Fill Value messages v1-v3 and the old
0x0004 message (validated against HDF5 2.0 output under default and latest
libver), builds a fully filled dataset when there is no storage, and writes the
fill value into exactly the chunk-grid cells absent from the chunk index —
never mistaking a stored zero for a hole, clipping edge chunks, any rank. It is
skipped entirely for the default (zero) fill value. The chunk index dispatch is
extracted from read_chunked_data into a reusable list_chunks.
The reader, lazy and mmap facades apply it on full reads; selection reads go
through a fill-aware full read when the fill value matters. h5py interop test
compares against h5py's own readback, including a sparse 2-D dataset and a
hyperslab straddling allocated and unallocated chunks.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
A dataset created from a committed (named) datatype stores only a shared-
message reference to it. The facade parsed those reference bytes as the
datatype itself, producing `Time { size: 0 }` and unreadable data, and an
attribute using a committed datatype was silently dropped.
- shared_message::parse_shared_ref had the encoding wrong: it skipped six
reserved bytes for version 2 (only version 1 has them) and had the version 3
types inverted (1 is the SOHM heap, 2 is "committed, in another object
header"). Verified against h5py 3.16 / HDF5 2.0, which writes
`02 02 <address>` under both default and latest libver bounds. Resolution
now dispatches on which field the reference carries.
- New shared_message::message_data resolves a header message through the
indirection; the reader, lazy and mmap facades use it for datatype,
dataspace and filter-pipeline messages.
- AttributeMessage honours the v2/v3 flags (bit 0 datatype shared, bit 1
dataspace shared) via the new parse_in_file, used everywhere file data is
available. Parsing a shared attribute without file access is now
FormatError::UnresolvedSharedMessage instead of a garbage datatype.
- h5py interop test covering both libver settings.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
- Dataset::filter_pipeline() (reader, lazy and mmap variants) swallowed parse
errors with `.ok()`, so a malformed pipeline message silently became "no
filters" and the still-compressed chunk bytes were returned as the data. It
now returns Result<Option<_>>; a present-but-unparseable pipeline is
Error::Format.
- FileBuilder::write used std::fs::write, which truncates the destination
first: a crash mid-write destroyed the existing file. It now writes a
sibling temp file, syncs it, renames it over the target and syncs the
directory, cleaning the temp file up on failure.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Dataspace and chunk dimensions are untrusted 64-bit fields, but the chunked
read paths computed `num_elements() as usize * elem_size` and
`chunk_dims.product() * elem_size` with plain arithmetic and fed the result to
`vec![0u8; n]`. A crafted file could wrap the product (under-sizing the output
buffer that chunks are then copied into) or request an allocation large enough
to abort the process.
- Dataspace::checked_num_elements, checked_byte_len, checked_chunk_byte_len
and alloc_output (try_reserve_exact) replace the plain products and
vec![0; n] at every chunked read site, plus the VDS and hyperslab paths.
Overflow and allocation failure are FormatError::Overflow.
- Dataspace::num_elements saturates instead of wrapping.
- A zero-element dataset returns early, which also keeps the stride products
in range when another dimension is huge.
- parallel_read.rs: the three `c_addr + size > len` bounds checks used a raw
add; they now use checked_add like the rest of the crate.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
test_hebbian_activation_boost failed intermittently. Root causes, all in the
query path:
- normalize_scores mapped a set of identical scores — including the
single-candidate case — to 0.0, so a lone perfect match contributed nothing
to the fused score. Identical positive scores now normalise to 1.0 (all
equally the best match); identical non-positive scores stay 0.0.
- merge_vector_keyword sorted a HashMap's entries by score alone and then
truncated, so which ties survived varied from run to run; hybrid_search had
the same problem in its final sort. Both now break ties by index.
- hybrid_search applied the Hebbian boost to every returned record, including
the zero-score filler that pads the list when fewer than k records match.
With random tie-breaking a filler record could collect as many boosts as the
real hit. Only records with a positive fused score are reinforced now.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
- ProvenanceStore::remap: compaction renumbers cache indices (which are the
provenance record ids) but nothing renumbered the ledger, so after any
compaction — including the automatic one in delete() — every surviving
record's hash was filed under a different record and the next
save_or_update raised a bogus High "integrity mismatch" alert.
- Pending anomaly alerts are capped (newest 1024 kept). Alerts never block a
save, and a session over its write limit alerts on every write, so a caller
that didn't drain them grew the queue without bound.
- WriteAnomalyDetector tracks at most 4096 sessions, forgetting the
least-active half on overflow instead of leaking one entry per session id
for the life of the process.
- snapshot() copies the pending WAL next to the .h5 copy, so a snapshot is
the store as it is now rather than as of the last checkpoint (it used to
silently omit up to wal_max_entries recent saves).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
- HDF5Memory::create/open take an exclusive advisory lock on <store>.h5.lock
(std File::try_lock, no new dependency). The store lives in memory and is
rewritten wholesale at each checkpoint, so two handles on one store used to
silently destroy each other's data; a second writer now gets
MemoryError::Locked. The OS drops the lock with the descriptor, so a crash
never leaves a stale lock. Acquisition retries for ~250 ms to absorb a
previous owner that is mid-teardown; AsyncHDF5Memory::shutdown releases the
lock once its writer task has stopped.
- HDF5Memory::open_read_only: a lock-free, point-in-time view (checkpoint +
current WAL contents, replayed in memory) that never writes — it does not
repair, upgrade or move the WAL, and anything that would persist returns an
error. The CLI's recall/stats/agents-md/export use it, so a store can be
inspected while an agent has it open. Tests that reopened a store purely to
verify on-disk state now use it.
- open() no longer fails on a WAL that cannot possibly be replayed (torn
header, bad magic): it is moved to <store>.h5.wal.corrupt-<ts>, reported via
HDF5Memory::quarantined_wal(), and the healthy .h5 opens from its last
checkpoint. A well-formed header with an unknown version still fails and is
left untouched — most likely a newer build's WAL, which must not be
discarded.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Eight MemoryConfig fields (float16, compression, compression_level,
compact_threshold, hebbian_boost, decay_factor, wal_enabled, wal_max_entries)
were never written to /meta, so reopening a store silently reset them to
defaults — a compressed store was rewritten uncompressed by the first
checkpoint after a reopen, and wal_enabled=false flipped back to true. They
are now stored as /meta attributes; each is optional on load so older files
keep opening with the previous defaults, and non-finite floats are ignored.
Writing the round-trip test exposed that `compression = true` never worked in
a default build: the embeddings dataset called with_zstd() unconditionally but
the agent crate never enabled the zstd feature, so every checkpoint failed
with "unsupported filter: 32015". The default build now compresses with
deflate (always available, pure Rust path); Zstd is opt-in via a new `zstd`
agent feature.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
A save_or_update that hit an existing record was logged as a plain Save, so
replaying the WAL appended a duplicate instead of updating in place. It is now
logged as WalEntryType::Update (0x04) carrying the target index, and replay
applies it with cache.update().
The WAL header version goes 3 -> 4 for the benefit of older binaries: they
don't know record type 0x04, would read it as a torn tail and truncate it and
everything after it. An unknown header version makes them refuse the file
instead. The framing is otherwise identical, so v3 files are read by the same
code and upgraded in place on open (the header is outside the CRC chain).
Also drop the redundant WAL truncate that several callers ran straight after
flush(), which already truncates.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
flush() writes the new .h5 and only then truncates the WAL. A crash in that
window left a .h5 that already contained the pending entries AND a WAL that
still listed them, and open() replayed the WAL unconditionally — every pending
entry came back twice.
A checkpoint now records a WalMark in /meta (wal_applied_len/wal_applied_crc):
the byte length and chained CRC of the WAL prefix it folded in. On open, if
the WAL's v3 CRC chain passes through exactly that position, the entries up to
it are skipped; otherwise (the normal case: the WAL was truncated) everything
is replayed. No WAL format change; files without the attributes behave as
before. WalFile tracks its chain length alongside running_crc and resumes both
on reopen.
Also make the checkpoint and snapshot durable as a unit: sync the temp file
before the rename and the parent directory after it, so a power loss can't
leave an empty or partial .h5 under the final name. This is per-checkpoint
cost only; individual WAL appends remain unsynced by design.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
The norms guard was the tautology `n.len() == n.len()`, so a norms dataset
of any length was trusted and corrupted every cosine score; other per-record
datasets were not length-checked at all, so a truncated file loaded and then
panicked on the first index. Mismatches are now MemoryError::Schema, stored
norms are used only when they match the record count, and embedding_dim == 0
with records present is rejected instead of panicking in chunks(0).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Formatting only. cargo fmt --check was already failing on main (accel SIMD
kernels, agent, format, migrate, bench); CI now enforces it.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
- clippy --all-targets plus a clawhdf5-format feature matrix (parallel, lz4,
zstd, pcodec, fast-checksum); fix the accumulated lint backlog in test,
bench and feature-gated code (no behaviour changes).
- Install python3 + h5py/numpy/netCDF4/xarray in the CI container and set
CLAWHDF5_REQUIRE_INTEROP=1, which makes a missing interop dependency a test
failure. Every h5py/netCDF4 interop test used to skip silently in CI. Run
the #[ignore]d writer_h5py_tests suite explicitly.
- cargo bench --no-run so benches can't rot; fix bench.rs and memory_bench.rs,
which no longer compiled against the current strategy/consolidation APIs.
- Optional fuzz smoke run via CLAWHDF5_FUZZ_SECONDS.
- CHANGELOG and docs/known-issues.md updated.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Every test created its own wgpu instance and device (with adapter-maximum
limits) concurrently, which could wedge the driver and hang the suite
indefinitely. Tests now hold a process-wide lock while they own a device, and
GpuAccelerator readback waits are bounded at 30s so a stuck driver surfaces as
GpuError::BufferMap instead of blocking forever.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Compound datasets written with default libver bounds (datatype message
version 1, i.e. plain h5py.File(path, 'w')) could not be read: the v1 member
layout has 28 bytes of legacy array fields after the byte offset
(dimensionality 1, reserved 3, permutation 4, reserved 4, four sizes 16) and
the parser skipped 24, so every following member was read 4 bytes off. v2 was
also wrong: it keeps the 8-byte name padding and has no array fields.
Found by adding a default-libver axis to the h5py-generated-file tests (HDF5
2.0 raised the default low bound to 1.8, so "default" files are a distinct
format path from libver='latest'). Adds byte-level v1/v2 regression tests, a
truncation test, and fuzz corpus seeds for v1 compound and native complex.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Bump all workspace crates, the node package and pyproject to 2.2.0 and
finalize the changelog. The repository URL in every manifest pointed at a
GitHub location that does not resolve; use the real origin.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Class 11 (datatype version 5) properties are a single base floating-point
datatype message, not a compound-style member list. The old parser read the
base type's bytes as member names, yielding a garbage datatype, and failed
with UnexpectedEof when a complex type was nested in a compound.
Parse the base type and surface the type as the equivalent {r, i} compound
(the shape h5py writes for numpy complex dtypes), with a size check against
the base type. Covered by byte-level tests taken from HDF5 2.0 output and an
h5py end-to-end test (writer_h5py_tests is now 27/27 against HDF5 2.0.0).
Found while validating a user report of InvalidDatatypeVersion
{ class: 6, version: 5 } against v2.1.0 (already fixed on main in a13ff51,
never released). Add docs/known-issues.md recording that report, this bug,
the open reference-v4 gap and a gpu_tests parallel-run hang; credit the
reporter in the changelog.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Two missions found largely disjoint ground — v3 in clawhdf5-ann, -io and
-migrate, v6 in -agent and -format — so this merge is mostly additive.
One genuine collision: BOTH runs independently implemented
`Dataset::verify_provenance` in clawhdf5/src/reader.rs, and git kept both,
producing `E0592 duplicate definitions`. They were functionally identical
apart from `self.file.data.as_bytes()` (v3) vs `self.file.as_bytes()` (v6).
Kept v6's — it is the variant that compiles against the current tree and
passed 52 suites, and its doc comment is the more honest one, stating both the
full-read cost and that an unkeyed hash stored beside its data is not a
tamper-evidence guarantee.
Verified present after the merge:
P1 clawhdf5-ann now depends on clawhdf5-accel; compute_distance calls
l2_distance / cosine_similarity instead of a scalar loop
P2 AsyncFileReader caches its handle and length behind a mutex
PR1 clawhdf5-migrate writes SHINES provenance, and validate checks it
v6 WAL torn-tail truncation, char-boundary truncate, agent hardening
52 suites pass, 0 failures.
Co-Authored-By: Claude Opus 5 <[email protected]>
`WalFile::open` scanned the chained entries to resume the CRC chain, then
seeked to END OF FILE to append. After a crash mid-append — the ordinary way a
WAL ends up damaged — that puts the next entry BEHIND the torn bytes:
[1..N verified][torn tail][N+1, chained to N]
`read_chained_entries` stops at the torn tail, so N+1 is unreachable forever
even though its `append` returned Ok and synced. Silent loss of an acknowledged
write, in the one situation a WAL exists for.
`read_chained_entries` now also returns the byte length of the verified prefix,
and `open` truncates to it and appends there. The torn tail was never
acknowledged to any caller, so discarding it loses nothing, and the file offset
then matches the `running_crc` the chain continues from.
Verified by negative control: with the previous `seek(End(0))` the new test
fails with "got 1 entr(y/ies) — the post-crash write was silently lost".
Introduced by neither this branch nor the chaining work — v2 seeked to EOF too.
What changed is that `open` now scans and therefore KNOWS where the verified
prefix ends, which is what makes the fix a two-line consequence of information
already in hand.
Co-Authored-By: Claude Opus 5 <[email protected]>
Mission: 01a00c41-bac0-7eb3-a8c8-8b7044f3086d
Phase: 01a00c41-bac2-71e3-a58b-c473421200ee
Committed by the ClawMates delivery pipeline from the agents' working tree. Authored by agents, not by the named committer.
than h5py (5e)
stable-worldmodel (arXiv 2605.21800, LeCun/Balestriero) supports HDF5 as
one of three native formats and measures generic HDF5 at 1,416-1,474
samples/s for per-frame sample loading. This measures clawhdf5 against
that shape, hardware-controlled: clawhdf5 and h5py reading the SAME file
on the SAME machine.
worldmodel_sampling example: mmap an (N,H,W,C) uint8 observation dataset,
read each frame once per pass in shuffled (dataloader) order. The file is
written by h5py (benchmarks/gen_worldmodel_frames.py) — clawhdf5 parsing
an externally-produced HDF5 file is itself the interop result — and read
by both clawhdf5 and the h5py counterpart (benchmarks/bench_worldmodel_h5py.py,
opening exactly stable-worldmodel's HDF5Dataset: swmr + 256 MB cache).
Results (tank, Ryzen 7 7800X3D, 20000x64x64x3 = 246 MB, in page cache,
median of 3):
clawhdf5 zero-copy view 593k samples/sec 8.1x
clawhdf5 materialised copy 518k samples/sec 7.1x
h5py (swmr, 256 MB cache) 73k samples/sec 1.0x
The materialised-copy row is the fair equal-work comparison (to_vec per
frame, matching h5py's numpy materialisation) and is still 7.1x faster;
that the copy costs almost nothing shows the gap is h5py's per-frame call
overhead, not data movement. Honest caveats in BENCHMARKS.md: absolute
numbers are NOT comparable to the paper's (different hardware, smaller
frames, no torch/transform), only the same-machine ratio is; this is an
in-page-cache measurement isolating read-path overhead, not disk
bandwidth.
Adds only an example, two benchmark scripts, and a BENCHMARKS.md section —
no library code. (Workspace clippy has pre-existing toolchain drift
unrelated to this change; tracked separately.)
Enables Rayon parallel compression for typical 4-chunk workloads (e.g.,
128×128 matrix with 32-row chunks). Rayon's dispatch overhead is ~2 µs,
worthwhile at ≥3 chunks with real compression work per chunk.
Previously the threshold was "> 4" which excluded 4-chunk datasets entirely
from parallel compression. Now "> 2" covers 3+ chunks.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
- Use Zstd level 3 instead of deflate(1) for embedding dataset compression.
Auto-shuffle (already the default since the TDT pre-filter commit) is now
the only shuffle needed — the explicit .with_shuffle() call was redundant.
- Benchmark: save_without_wal_single improves 67 → 61 µs (-9%).
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Implements WAL group commit optimizations (arXiv:2507.13062):
1. Serialize each WAL entry to a local Vec<u8> before writing, reducing
write() syscalls per entry from ~8 to 1.
2. Defer header entry_count updates to every GROUP_COMMIT_SIZE (8) entries
instead of per-entry, eliminating 3 lseek() + 1 write() per entry.
3. Fix read_entries() to read until EOF instead of looping entry_count
times — the header count is now a pre-allocation hint only. This is
strictly more robust: tolerates stale counts from deferred updates AND
truncated files from crashes mid-write.
Benchmark results:
- wal_flush_100_entries: -7.8% latency improvement (469 µs)
- save_with_wal_single: -1.7% (18.2 µs)
- save_without_wal_single: -2.3% (67 µs, full HDF5 write)
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Following arXiv:2506.18062 (TDT pre-filter) and matching h5py default
behavior: the shuffle filter is now automatically applied before any
compression codec (deflate, Zstd, LZ4, Pcodec) unless explicitly
disabled with .without_shuffle().
Benchmark results (f32 matrices, shuffle+codec vs unshuffled baseline):
- Zstd-3 at 512×512: 610 → 764 MiB/s (+25%)
- Deflate-6 at 128×128: 132 → 401 MiB/s (+204%)
- Deflate-6 at 512×512: 280 → 745 MiB/s (+166%)
Both codecs now reach parity at ~750 MiB/s for large matrices.
Changes:
- Add no_shuffle field to ChunkOptions (opt-out via .without_shuffle())
- Auto-add FILTER_SHUFFLE in build_pipeline() when compression is active
- Add DatasetBuilder.without_shuffle() method
- Update pipeline tests to reflect new 2-filter default
- Add chunk_options_pipeline_deflate_no_shuffle test
- Update BENCHMARKS.md with measured throughput improvements
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Implements Pcodec (filter ID 32023) via the `pco` 1.0.x crate as a new
optional compression codec. Pcodec achieves 30–94% better compression
ratio than Zstd for f32/f64 columnar data at 1–5 GiB/s decompression
speed, making it ideal for write-once/read-many embedding archives.
Write throughput at 512×512: 591 MiB/s (parity with Zstd-3 at 610 MiB/s).
For smaller chunks Zstd-3 remains faster due to Pcodec's fixed per-chunk
distributional analysis overhead.
- Add FILTER_PCODEC = 32023 constant to filter_pipeline.rs
- Add pcodec_compress/pcodec_decompress using pco::standalone API
- Wire into compress_chunk/decompress_chunk dispatch
- Add ChunkOptions.pcodec field and DatasetBuilder.with_pcodec() method
- Enable pcodec as highest-priority codec in build_pipeline()
- Add pco dep (optional, feature = "pcodec") to clawhdf5-format/clawhdf5
- Add write_2d_chunked_pcodec benchmark comparing pcodec vs zstd-3
- Document results in BENCHMARKS.md
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Add features = ["zstd"] to clawhdf5-bench dev-dependency so the
write_2d_chunked_zstd benchmark no longer panics with UnsupportedFilter(32015).
Update BENCHMARKS.md and README.md with measured results from the full
h5bench write suite (2026-06-30, post write-performance improvements):
- Zstd-3 hits 593 MiB/s at 512×512 vs deflate-6's 280 MiB/s (2.12×)
- Zstd-3 hits 330 MiB/s at 128×128 vs deflate-6's 132 MiB/s (2.51×)
- Sequential f64 batch write improved ~8-11% from owned-Vec IO path
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Four independent write-path improvements:
1. Cache compressed chunks between Pass 1 and Pass 2 (chunked_write.rs,
file_writer.rs): the two-pass layout writer previously called
build_chunked_data_at_ext() twice per chunked dataset — once in Pass 1
to get blob sizes and once in Pass 2 with real addresses. Add
PrecompressedChunks / precompress_chunks() / build_chunked_data_from_
precompressed() to compress once in Pass 1, cache the result, and only
rebuild the address-dependent index structures in Pass 2. Expected
~2× speedup for chunked+deflate writes (512×512 deflate: 3.33ms → ~1.7ms).
2. SIMD-vectorisable shuffle filter (filters.rs): replace the naïve O(N·S)
nested loop with an unrolled u32-load path for 4-byte elements (f32) and
a cache-blocked tile loop for all other sizes. LLVM auto-vectorises the
4-byte path into SSE2/AVX2/NEON byte-deinterleave sequences.
3. Zstd benchmark variant (h5bench_write.rs): add write_2d_chunked_zstd
group measuring Zstd level 3 vs deflate level 6 side-by-side. Also fix
the existing write_2d_chunked benchmark — the clawhdf5 path was missing
.with_deflate(6), making the comparison apples-to-oranges. Add arXiv-
backed doc recommendation on DatasetBuilder::with_zstd().
4. Zero-copy HNSW save (hnsw.rs, clawhdf5-io/lib.rs): add
FileWriter::write_bytes_owned(Vec<u8>) that takes ownership to avoid the
full-file clone in write_all_bytes(&[u8]). HNSW::save_to_hdf5 uses it.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Full head-to-head results from Criterion suite (100 samples each):
sequential read/write, chunked write + deflate, metadata ops, group
traversal. Includes interpretation section explaining the structural
reasons for each gap.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Critical fixes from whole-branch code review:
- libaec-sys: fix flag constants to match <libaec.h> exactly
(PREPROCESS=8, MSB=4, RESTRICTED=16; drop non-existent AEC_ALLOW_K13)
and add aec_buffer_encode FFI declaration
- filters_szip: fix cd index for bits_per_sample (cd[2] per H5Z_SZIP_PARM_BPP,
not cd[4]); fix option-mask mapping (NN=0x20, MSB unconditional); add two
real encode→decode roundtrip tests (no-NN and NN) that exercise libaec end-to-end
- file_writer: fix serialize_vds_mappings to delegate to data_layout_write
(eliminates the buggy duplicate that always emitted version=1 even for
external-file mappings); retains trailing Jenkins checksum
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
The previous aec_buffer_decode declaration used flat parameters which
don't match the actual libaec C API; this caused a SIGSEGV at runtime.
Replace with the correct aec_stream struct (mirroring <libaec.h>) and
update filters_szip.rs to populate and pass &mut AecStream.
Also add empty-input guard in szip_decode_impl and fallback library
path search in build.rs for distros that omit the .pc file.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Adds three Criterion benchmark suites mirroring the h5bench HPC I/O
benchmark workloads in pure Rust — no C libhdf5 required for the default
path, with an optional `libhdf5-compare` feature for side-by-side numbers.
- benches/h5bench_write.rs: write_1d_contiguous, write_2d_chunked,
write_f64_batch, write_multi_dataset, write_with_attrs
- benches/h5bench_read.rs: read_sequential, read_f64_sequential,
read_chunked_2d, read_from_disk, read_hyperslab
- benches/h5bench_meta.rs: metadata_attrs_write, metadata_attrs_read,
metadata_groups_create, metadata_groups_traverse, metadata_string_attrs
All benchmarks pass `cargo bench --bench <name> -- --test` and clippy
reports zero warnings. Run with `cargo bench -p clawhdf5-bench`.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Add `virtual_sources: Option<Vec<VdsMapping>>` field and `with_virtual_sources()` method to `DatasetBuilder`. In `FileWriter::finish()`, VDS datasets skip raw-data storage and instead serialize their source mappings into a global heap collection (version-1 same-file encoding) referenced by an HDF5 v4 layout-class-3 message. The two-pass address-computation loop handles VDS in both passes: pass 1 computes the fixed-size OH and pre-builds the heap blob; pass 2 places the blob at the correct file offset and rebuilds the OH with the real global heap address. Three new tests verify: (a) same-file two-source round-trip with mapping verification, (b) external-file source encoding, and (c) empty mapping list.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
clawhdf5-migrate:
- Real content validation: the post-migration check reads the written HDF5
back (new hdf5_reader) and compares actual content — chunk text, embeddings,
and every session/entity/relation field — to the source, not just row counts.
A representative sample of chunk rows is verified by default; --validate-full
checks every row. A count-preserving corruption no longer passes.
- Configurable schema: SQL is built from a SchemaConfig (table + ordered column
names, defaulting to the ZeroClaw layout) instead of hardcoded queries, with
--chunks-table / --sessions-table / --entities-table / --relations-table.
- Streaming count pass: --dry-run does a COUNT(*)-only pass per table instead
of loading every row.
- Incremental migration: --incremental reads the existing output, reads only
source chunks with id greater than the last migrated id, and appends them
(metadata groups refreshed from source) rather than re-migrating everything.
clawhdf5-format:
- read_as_f32 / read_as_f64 now decode IEEE-754 half-precision (2-byte) floats
via a no_std-safe bit conversion — needed to read float16-stored embeddings
back (e.g. for migrate's content validation), previously a TypeMismatch.
Tests: f16 read unit test; migrate tests for content-corruption detection,
custom table names, and incremental append; CLI smoke-tested end-to-end and the
dense/incremental output verified with h5py.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Dense attribute and dense link storage capped at a single fractal-heap direct
block (~64 KiB of heap data — a few thousand objects); beyond that the writer
produced an invalid oversized block. Lift the cap with a root indirect block.
When the serialized objects don't fit in one direct block, build a root
indirect block (FHIB) over multiple direct blocks sized by the doubling table
(start 512, width 4, doubling per row up to 64 KiB). Objects are packed
row-major across blocks, each block carries its logical block offset, and heap
IDs encode each object's heap offset (block offset + position). The FRHP points
root -> FHIB with the row count; unused slots in the current rows are undefined.
The fractal-heap builder is unified: FractalHeapBlock now carries the full heap
blob, and a shared write_frhp helper serializes the header for both the
single-block and multi-block paths. The single-block path is unchanged
(byte-identical), so existing dense attrs/links stay valid.
Validated end-to-end: a 2,500-attribute object and a 2,500-link group round-trip
through our reader and are read correctly by h5py. Objects still may not span a
block (no huge-object path).
Tests: facade round-trips for multi-block dense attrs and dense links, plus an
h5py-gated interop test (verified against the real h5py environment).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The fractal-heap reader split direct vs indirect block rows using the FRHP
"Starting # of Rows in Root Indirect Block" field (a constant, typically 1),
mislabeled as starting_row_of_indirect_blocks. For any heap whose data spans
more than one direct block — common in libhdf5 files with a large group or
many dense attributes — this treated direct blocks as indirect and walked into
garbage, failing with InvalidFractalHeapSignature.
Derive the split from the heap geometry instead: max_direct_rows =
log2(max_direct_block_size / starting_block_size) + 2. Rows below it hold
direct blocks; rows at/above hold child indirect blocks.
Validated against an h5py-written group with 400 dense attributes (root
indirect block, 4 rows, 13 direct blocks): all values now read correctly.
Regression fixture covers an 80-attribute multi-block heap.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
A group with more than 8 links (libhdf5's compact max_compact default) is now
written densely instead of as inline Link messages: the links live in a
single-direct-block fractal heap indexed by a v2 B-tree of type 5 (link-name
index), referenced from the group's LinkInfo message. This matches libhdf5's
compact->dense switchover and keeps large groups out of the object header.
- Extract the byte-identical fractal-heap builder shared by dense attributes
and dense links, parameterized by heap_id_length / max_heap_size. Attributes
keep 8 / 40; links use 7 / 32 to match libhdf5 (reverse-engineered: an
h5py-written dense group uses heap_id_length 7, max_heap_size 32, type-5
record = hash(4) + heap_id(7) = 11 bytes). This was the cause of an initial
"object overruns end of direct block" error from h5py.
- build_group_oh gained an optional dense LinkInfo (omitting inline Link
messages); the two-pass file assembly allocates each group's link blob after
its object header and rebuilds it with real target addresses in the final
pass (link-message size is address-independent, so layout is stable).
Validated end-to-end: our reader round-trips dense groups, h5py reads the
groups we write, and dense attributes remain byte-identical (still h5py-valid).
The agent's 9-dataset memory group now writes densely and round-trips. Single
direct block only (~a couple thousand links); indirect blocks remain a TODO.
Tests: facade round-trip (20-link dense group + compact sibling) and an
h5py-gated interop test confirming libhdf5 reads our dense groups.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The readers added this cycle parse untrusted bytes, so malformed/hostile
input must produce errors — never a panic, OOM, or unbounded recursion.
Audited each new surface and fixed the concrete vectors, each covered by an
adversarial regression test:
- Paged Fixed Array: `1 << max_nelmts_bits` shift overflow (u8 up to 255);
element-count bounded by file size; element/page offset multiplies checked.
- H5S selection decoder: ALL/NONE validate they have the 16 bytes they claim
to consume; hyperslab rank capped at 32 (H5S_MAX_RANK); iter_linear
coordinate/stride/product arithmetic uses checked ops.
- VDS mapping parser: drop pre-allocation from the untrusted `nused`;
bounds-check all selection slicing.
- scale-offset / N-Bit filters: `1 << minbits` overflow at minbits==64; N-Bit
`bit_offset + precision` overflow; N-Bit type-tree recursion depth capped to
stop a crafted nested tree from overflowing the stack; element counts bounded
by the chunk's expected decompressed size (threaded the previously-unused
chunk_size into both decoders) so a bogus count can't over-allocate.
- VDS assembly: a virtual dataset whose source is itself virtual (a cycle) now
errors instead of recursing into a stack overflow.
16 new adversarial tests; full format suite (482 lib) + agent + facade green;
clippy clean.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
clawhdf5-agent: fixed-length string datasets (memory text chunks, session
summaries, ids, tags, entity/relation names) were stored uncompressed behind
a stale "chunked compound not yet supported" comment. Chunked writes work for
fixed-size string/compound datatypes like any other, so write_string_dataset
now chunks + deflates once a dataset's payload reaches 4 KiB — large,
redundant NullPad content compresses well while tiny metadata stays
contiguous (no chunk-overhead bloat). The dead `compress` parameter is
removed in favor of this size heuristic.
clawhdf5-format: enabling string compression exposed a latent bug — the
per-file ChunkCache built its chunk index once and reused it for every
chunked dataset in the file, keyed only by chunk coordinate with no dataset
discrimination. With one chunked dataset per file this never surfaced; with
two of different rank (a 1-D compressed string array and the 2-D embeddings
matrix) the first dataset's rank-1 index was reused for the second, panicking
with an out-of-bounds chunk coordinate. The cache now binds to a dataset by
its chunk-index address and rebinds — dropping the index, chunk-index map,
layout, and decompressed slots — whenever the dataset being read changes,
while still caching repeated/sequential access to the same dataset.
Tests: facade regression reading a 1-D compressed string dataset and a 2-D
compressed f32 dataset through one shared File cache (verified to panic
without the fix); existing agent e2e tests (large text chunks, migration
round-trip) now pass with compression on.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The HDF5 library does not implement the scale-offset filter's floating-point
E-scale mode. When asked for it (cd_values[0] = 1) it stores the chunk raw
(no minbits/minval header) and sets the chunk filter mask to skip the filter,
so such datasets read back verbatim purely by honoring the per-chunk filter
mask — no E-scale decoder is required.
Add a fixture written via the HDF5 low-level API (exact-representable values)
and a test asserting it reads back verbatim, locking in the filter-mask path.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
VDS sources living in other files were previously unsupported because the
pure-byte read API has no filesystem. Add a resolver seam and wire a default.
clawhdf5-format:
- Add VdsSourceResolver (Fn(&str) -> Option<Vec<u8>>) and
read_raw_data_full_with_resolver. read_virtual_data uses the resolver to
fetch an external source file's bytes by its stored name, then reads the
named source dataset from those bytes and scatters as usual. A resolver
returning None leaves the region at fill (HDF5's missing-source behavior);
an external source with no resolver at all is a clean error. read_raw_data_full
is unchanged (delegates with no resolver).
clawhdf5:
- File now records the directory it was opened from and, for virtual layouts,
reads through a default resolver that loads sibling source files relative to
that directory. So File::open(virt).dataset(d).read_*() transparently
assembles cross-file VDS. In-memory files (from_bytes) have no directory, so
only same-file VDS resolves there.
Tests: format-layer external read with an injected resolver (and the
no-resolver error path), plus facade tests that drop both files in a temp dir
and read through File::open — covering successful resolution and the
missing-source-is-fill case.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Generalize Selection iteration from 1-D to arbitrary rank: iter_linear(dims)
enumerates a selection's row-major linear indices over a dataspace of the
given shape (ALL, NONE, regular hyperslabs, points), which is the order HDF5
uses to pair virtual and source selections.
read_virtual_data now passes the full virtual/source dimensions instead of a
single extent, so multi-dimensional block mappings scatter to the correct
non-contiguous linear positions. read_named_dataset_raw returns the source
dataset's dimensions. The rank-1 restriction is removed; only external-file
sources remain unsupported.
Tests: 2-D integration fixture (vds_2d_same_file.h5: two 2x2 sources placed
as non-contiguous blocks in a 4x4 virtual) plus N-D iter_linear unit tests
(block, strided, ALL, rank-mismatch). The 1-D path is unchanged.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
A virtual layout previously returned UnsupportedVersion. Implement reading
for the common 1-D, same-file case, reverse-engineered and validated against
HDF5 2.0.
- Rewrite parse_vds_mappings to the real global-heap block format
(version(1) · nused(length_size) · entries · checksum(4)), where each
entry is source-file(null) · source-dataset(null) · source-selection ·
virtual-selection. Block version 1 encodes a same-file source as a single
0x04 marker in place of the file name; version 0 stores an explicit file
name. The selections are H5S-serialized and self-describing in length, so
they are decoded to find entry boundaries. The previous parser used a
guessed layout that did not match real files.
- Extend Selection with decode_serialized() (H5S_select_serialize: ALL,
NONE, and version-3 regular hyperslabs) and iter_linear_1d().
- Add read_virtual_data: resolve the mapping block from the global heap,
read each same-file source dataset, and scatter its selected elements into
the virtual buffer; unmapped regions stay at the zero fill value.
External-file sources and N-D selections return a clean unsupported error.
Tests: real-file integration test (vds_same_file.h5: partial source slice +
fill gap), selection decoder unit tests built from the fixture bytes, and
same-file/external mapping-parser unit tests.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
A filtered, fixed-dimension dataset with more than one Fixed Array
data-block page (>1024 chunks by default) previously failed with
"paged Fixed Array data blocks not yet supported".
Implement the paged data-block layout, reverse-engineered and validated
against an HDF5 2.0 file:
- after the FADB prefix: a page-init bitmap (one bit per page, MSB-first
within each byte), a 4-byte checksum, then the pages;
- each page is a fixed full-size slot of page_nelmts elements plus a
4-byte checksum, with only the final page shorter;
- uninitialized pages still occupy their slot (zero-filled), so the
bitmap — not a 0xFF sentinel — marks a whole page unallocated.
Element parsing is factored into parse_fa_element, shared by the
non-paged and paged paths.
Tests: real-file integration test against a minimal 2-page gzip fixture
(v4_fixed_array_paged.h5) plus a synthetic unit test covering a
multi-byte/MSB-first bitmap, a skipped uninitialized page, and a short
final page.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The typed read paths (read_as_i32/i64/u64/f32/f64) rejected Array datatypes
with a TypeMismatch, so an array-typed compound member (common with N-Bit /
reduced-precision data) could not be read. They now unwrap an Array to its base
type and read the flat sequence of base elements, recursing for nested arrays.
Base-type precision rules (e.g. reduced-precision sign extension) apply to the
elements.
Validated end-to-end against an HDF5 2.0 compound with an array member: the
array field reads [-1, 100, 1000, -32768] with correct 16-bit sign extension.
Adds a regression test for flat and nested array reads.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Generalizes the N-Bit filter (id 5) decoder from atomic-only to the full
recursive type tree carried in the filter client data: atomic
([1, size, order, precision, offset]), array ([2, total_size, <base>]) and
compound ([3, total_size, nmembers, (offset, <node>)*]), nestable to any depth.
The decoder parses the tree once, then walks it per element with an MSB-first
bit reader, placing each leaf field's significant bits at its byte/bit offset in
a zero-filled element — HDF5's canonical layout. Float members are encoded as
full-precision atomics and handled transparently. Validated end-to-end against
HDF5 2.0 / h5py: compound int+int, compound with an array member, and compound
with a float member all decode to the exact canonical bytes. Adds h5py-free unit
tests from real captured chunks. (Reading array-typed compound *fields* into a
flat buffer is a separate datatype-reader concern.)
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
HDF5 stores a fixed-point value whose datatype precision is smaller than its
storage size zero-filled above the precision; the sign of a reduced-precision
signed integer lives in the precision field, not the storage word, and is
applied during datatype conversion. clawhdf5 previously read the full storage
word, so e.g. a 16-bit-precision -1 (stored 0x0000ffff) read as 65535.
The integer read paths (read_as_i32/i64/u64/f32/f64) now extract the
[bit_offset, bit_offset+bit_precision) field and sign-extend (signed) or mask
(unsigned). Full-width types are unchanged — the bulk-copy fast paths are gated
to full width, so the common case keeps its memcpy and behaviour.
This completes signed N-Bit reads (now exact end-to-end) and also fixes
un-filtered reduced-precision signed/unsigned integer datasets. Validated
against HDF5 2.0 / h5py; adds h5py-free regression tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Implements decompression for the N-Bit filter (id 5), atomic integer/float
variant — previously returned UnsupportedFilter(5). N-Bit packs each element's
significant `precision` bits MSB-first with no header; decode reads those bits
per element and places them at the datatype's bit offset in a zero-filled
`size`-byte element, reproducing HDF5's canonical reduced-precision layout
(verified byte-for-byte against the equivalent un-filtered dataset).
Reverse-engineered and validated against HDF5 2.0 / h5py: unsigned
reduced-precision datasets now read end-to-end with exact values. Signed
reduced-precision values are restored to their canonical (zero-filled) bytes;
sign-extending them to the application width is the datatype reader's job — a
pre-existing concern shared with un-filtered reduced-precision data. Recursive
compound/array N-Bit layouts remain unsupported. Adds h5py-free unit tests from
real captured chunks.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Extends the scale-offset filter (id 6) decoder to the floating-point D-scale
variant (H5Z_SO_FLOAT_DSCALE) alongside the integer variant. Shares the header
parsing and MSB-first code unpacking; reconstruction is
`value = minval + code / 10^scale_factor`, where `minval` is the minimum float
stored in the header and the all-ones code is the (defined) fill value.
Reverse-engineered and validated against HDF5 2.0 / h5py across f32 and f64,
negatives, decimal scale factors D=1..5 and multi-chunk datasets (decoded values
match h5py to full precision). The float E-scale variant remains unsupported.
Adds h5py-free unit tests from real captured chunks.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Implements decompression for the scale-offset filter (id 6), integer mode
(H5Z_SO_INT) — previously returned UnsupportedFilter(6). The on-disk format was
reverse-engineered against HDF5 2.0 / h5py and verified across signed/unsigned
element sizes, negative minima, multi-chunk datasets and fill-value handling:
minbits (u32 LE) | 0x08 | minval (8 bytes LE) | 8 reserved bytes |
MSB-first packed codes (nelmts * minbits bits)
Each code is `value - minval`; the all-ones code is reserved for the (defined)
fill value. The floating-point variants (D-scale/E-scale) use a different
algorithm and remain reported as unsupported.
Validated end-to-end (a 200-element chunked scale-offset dataset, plus negative
and unsigned datasets, now decode to the exact h5py values). Adds h5py-free unit
tests using real captured compressed chunks.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Follow-up to the v5 compound fix, found by an interop sweep over diverse
h5py/HDF5 2.0 (libver=latest) datasets:
- Array datatype (class 10) version 5 was rejected. v3/v4/v5 share the same
array encoding, so the parser now accepts 3-5.
- Data Layout message version 5 was rejected, which broke EVERY chunked/
compressed dataset written by modern HDF5. v5 reuses the v4 message
structure, so it now routes through parse_v4.
Validated end-to-end: a gzip-compressed, Fixed-Array-indexed v5 chunked dataset
now decodes to the correct values. Adds h5py-free regression tests using the
real v5 array-datatype and chunked-layout bytes, and updates the layout
invalid-version test to use v6.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
clawhdf5 rejected datatype message version 5 for the compound class with
"invalid datatype version 5 for class 6", so it could not read compound
datasets written by modern HDF5 / h5py with libver=latest. v5 reuses the same
compact member encoding as v3/v4 (name, variable-width offset, member type), so
the parser now accepts versions 3-5 for compound.
Found by running the previously-ignored h5py interop tests against h5py 3.16 /
HDF5 2.0. Adds an h5py-free regression test using the real v5 datatype bytes.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Records the parallel chunk-compression perf change and the doc sweep that
landed after the v2.1.0 tag.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
- Add the `hnsw` flag (default-on) to the agent feature table and the
fast-deflate/system-zlib/fast-checksum/lz4/zstd/blake3 flags to the format
table.
- Add entity_extract and async_memory to the agent module overview.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
build_chunked_data_at_ext now compresses all chunks via compress_all_chunks
(previously dead code) before laying them out, so compression runs across
rayon threads under the `parallel` feature when there are >4 filtered chunks.
Layout is unchanged — compression preserves chunk order, so on-disk bytes are
identical to the sequential path. The agent crate enables `parallel`, so this
speeds up compressed embedding writes.
Removes the #[allow(dead_code)] on compress_all_chunks and gates
PARALLEL_COMPRESS_THRESHOLD behind the `parallel` feature.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Sweep of the docs after the v2.0.0 rename and recent changes:
- Per-crate READMEs (13 files): rename leftover rustyhdf5-*/edgehdf5-*
package names and badges to clawhdf5-*, bump usage versions to 2.1.0.
- README: update stale test badge (417 -> 1500+), workspace stats
(15 crates/72K -> 17 crates/84K), agent crate stats (20.7K/32 modules),
and add the missing clawhdf5-napi and clawhdf5-bench crates to the tree.
- CLAUDE.md: correct the CLI subcommand list (inspect/dump/index/search ->
the actual create/save/search/recall/stats/flush-wal/agents-md/export/snapshot).
No code changes. Verified there are zero todo!()/unimplemented!() macros and
no TODO/FIXME comments in the tree.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
