quantumclaw/clawhdf5
3
1
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases 1 Wiki Activity

Compare commits

base: quantumclaw/clawhdf5:v2.4.0
Branches Tags
quantumclaw/clawhdf5:main quantumclaw/clawhdf5:feat/p3-remote-editor quantumclaw/clawhdf5:feat/p2-perf-coverage quantumclaw/clawhdf5:feat/p1-conformance-report quantumclaw/clawhdf5:fix/phase0-correctness quantumclaw/clawhdf5:docs/withdraw-zeroclaw quantumclaw/clawhdf5:chore/remove-agent-feature
quantumclaw/clawhdf5:v2.7.0 quantumclaw/clawhdf5:v2.6.0 quantumclaw/clawhdf5:v2.5.0 quantumclaw/clawhdf5:v2.4.0 quantumclaw/clawhdf5:v2.3.0 quantumclaw/clawhdf5:archive/clawmates-mission-01a00cfa-c69f39fd-i2 quantumclaw/clawhdf5:archive/clawmates-mission-01a00c57-4c1f211e quantumclaw/clawhdf5:archive/clawmates-mission-01a00bbb-haiku-baseline quantumclaw/clawhdf5:archive/clawmates-mission-019ff5b1-1d1c5b0f quantumclaw/clawhdf5:v2.2.0 quantumclaw/clawhdf5:pre-rewrite-2026-06-05 quantumclaw/clawhdf5:v2.1.0
...
compare: quantumclaw/clawhdf5:feat/p3-remote-editor
Branches Tags
quantumclaw/clawhdf5:feat/p3-remote-editor quantumclaw/clawhdf5:main quantumclaw/clawhdf5:feat/p2-perf-coverage quantumclaw/clawhdf5:feat/p1-conformance-report quantumclaw/clawhdf5:fix/phase0-correctness quantumclaw/clawhdf5:docs/withdraw-zeroclaw quantumclaw/clawhdf5:chore/remove-agent-feature
quantumclaw/clawhdf5:v2.7.0 quantumclaw/clawhdf5:v2.6.0 quantumclaw/clawhdf5:v2.5.0 quantumclaw/clawhdf5:v2.4.0 quantumclaw/clawhdf5:v2.3.0 quantumclaw/clawhdf5:archive/clawmates-mission-01a00cfa-c69f39fd-i2 quantumclaw/clawhdf5:archive/clawmates-mission-01a00c57-4c1f211e quantumclaw/clawhdf5:archive/clawmates-mission-01a00bbb-haiku-baseline quantumclaw/clawhdf5:archive/clawmates-mission-019ff5b1-1d1c5b0f quantumclaw/clawhdf5:v2.2.0 quantumclaw/clawhdf5:pre-rewrite-2026-06-05 quantumclaw/clawhdf5:v2.1.0
421 Commits
v2.4.0 ... feat/p3-remote-editor
Author SHA1 Message Date
osobhandClaude Opus 5.5 ef428d756c format: slice entry points for the facade's hot *_in calls (local listing back to main's speed)
CI / test-arm64 (pull_request) Successful in 1m25s
Details
CI / test (pull_request) Successful in 19m0s
Details 
Since the M2 merge the facade handed in-memory files to the generic
`*_in` parsers as `&[u8]` (`with_bytes!`), which instantiates them in
the facade crate, where the format crate's private helpers do not
inline without LTO: listing a 400-group v1 file through `File::open`
was 7-10% slower than main. `ObjectHeader::parse_in`,
`group_v2::{resolve_child_in, resolve_group_children_in,
resolve_path_any_in}` and `attribute::{extract_attributes_tolerant_in,
find_attribute_in}` now pass a storage with `as_contiguous()` to their
non-generic slice entry point, compiled once in the format crate; other
storages reach the same generic core as before.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 22:28:22 -05:00
osobhandClaude Opus 5.5 4313917b4d format: object headers without per-header allocations for their chunk list 
Reading continuation chunks from a queue (7e5e920, a69c5be) allocated a
queue Vec and a BTreeSet of chunk starts for every header, and inlined
the per-chunk message loop into the generic parser: ObjectHeader::parse
over 401 version-1 headers went from 24.8 to 45.7 us.

ChunkSpans now keeps the first 8 chunks in an inline array (cycle check
by scan) and is also the read queue; only a header of more chunks
allocates (a boxed spill list and start set). The message loop of one
version-1 chunk is its own non-generic function. Same checks as before:
any number of chunks up to 65,536, cycles refused, chunks bounded by the
file size, one chunk buffer alive at a time, libhdf5 message order,
overlap allowed. The cycle test now also covers spilled chunk lists.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 20:30:16 -05:00
osobhandClaude Opus 5.5 011e0dbb96 docs: conformance report after range-read M2/M3 and editor coverage (600 of 697 ok) 
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 19:34:27 -05:00
osobhandClaude Opus 5.5 f37e7ae326 format: bound ZFP chunk fetches at 4x, not n + n/4 
The M2 fetch bound gave every codec n + n/4 + 4096 stored bytes, but
ZFP's fixed-rate mode stores up to 64 bits per value, doubling 4-byte
types: 60 of the 2205 zfp_interop datasets (rate 64, f4/i4) failed with
'compressed stream ends early' once M2 and ZFP were merged.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 19:33:03 -05:00
osobh 7447dce121 Merge branch 'feat/p3-editor-coverage' into feat/p3-remote-editor 
# Conflicts:
#	CHANGELOG.md
#	CLAUDE.md
#	docs/design/range-reads.md
 2026-09-26 19:17:46 -05:00
osobh 93e2d5f365 Merge branch 'feat/p3-m3-remote' into feat/p3-remote-editor 
# Conflicts:
#	crates/clawhdf5/tests/storage_equivalence.rs
 2026-09-26 19:17:32 -05:00
osobh 2893b6c974 Merge branch 'feat/p3-m2-raw-data' into feat/p3-remote-editor 2026-09-26 19:17:13 -05:00
osobhandClaude Opus 5.5 ea0508aaa5 format: no truncating u64 -> usize casts in gather_storage and ExtentBytes 
check-32bit-casts.sh flagged two casts added by the previous commits; both
values are bounded (checked by gather_storage's first walk, and built from
a usize fetch length), so they go through addr::saturating_usize.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 19:05:45 -05:00
osobhandClaude Opus 5.5 75444950f3 clawhdf5: storage harness compares errors, not just failures 
The facade equivalence harness turned every data-read error into "Err", so
it could not see File::open_storage failing differently from File::open
(a Storage or ContiguousStorageRequired error where the mmap path gives a
decode error, say).

- value() keeps the whole error. The only allowance is for a line on which
  File::open itself varies between opens — the chunk cache lists a damaged
  dataset's chunks in hash-map order, so which failing chunk a full read
  reports varies (cve-2025-2310.h5, the one corpus file where this shows):
  both sides must fail there, and a fresh File::open (up to 64) must
  reproduce the storage's exact error. Open errors were already compared
  in full; they still agree.
- The storage transcript may not contain ContiguousStorageRequired.
- More selections: a strided hyperslab (every third row) through
  read_f64_selection, and out-of-order points through read_selection and
  read_i64_selection.
- harness_compares_errors_not_just_failures checks the harness itself:
  two different errors are different values, and a difference File::open
  does not produce is reported.

With full errors the harness passes on the 61 fixtures and on the corpus
(701 files, 621 open).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 19:04:56 -05:00
osobhandClaude Opus 5.5 8236b0e30a edit: skip heap blocks too small for an attribute, as libhdf5 does 
An attribute needing a heap block larger than the next one was refused
("skipping blocks too small for an object", "a first object too large for
the starting block"); once an object's move to dense storage was refused
it refused every new attribute, so 24% of set_attr calls in the review's
random workload failed.

Following H5HF__hdr_update_iter, H5HF__man_iblock_root_create/_double and
H5HF__hdr_skip_blocks, the smaller blocks are now skipped: the iterator
moves past them and they become an indirect free section with a first
row section (serialized, class 1, as H5HF__sect_indirect_serialize writes
it) and ghost normal rows, added as returned space so it merges with a
range skipped just before it (H5HF__sect_indirect_merge_row). Later
objects that best-fit a row section get a block created there
(H5HF__man_iblock_alloc_row / H5HF__sect_indirect_reduce_row: from the
start or end of the range, or from its middle, which splits it, with
libhdf5's span bookkeeping). Heaps with such sections, as libhdf5 writes
them, are now read too (they were refused at open).

dense_skipped_blocks_match_libhdf5 drives every path (merge, split, end,
last entry, row wrap) on earliest/v110/latest files against libhdf5
doing the same edits one session each; heaps, free sections and index
B-trees are equal after every phase. The refusal test now checks the
skip against libhdf5 and keeps a real refusal (last object in a block);
clawhdf5-written heaps get 1-4 KiB attributes too. The three tests fail
on the previous fheap.rs. Random workload refusals: 24% -> 2.2%, all the
documented last-object-in-a-block case.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 18:50:23 -05:00
osobhandClaude Opus 5.5 c2ae7846c9 docs: label the h5repack sizes in the append-waste figures 
306 104 and 49 930 are h5repack of the editor's file; the text read as if
they were h5repack of libhdf5's, which measures 305 954 and 50 188. Both
are now given, from measure_append_waste rerun on 2026-09-26 (file sizes
unchanged).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 18:50:23 -05:00
osobhandClaude Opus 5.5 a69c5be8b2 format: read object header chunks from a queue, one buffer at a time 
The version-1 chunk walk nested continuation chunks depth-first and kept
every enclosing chunk's buffer alive, up to 65 536 chunks. With storage
that hands out owned buffers (CountingStorage, the Storage trait, remote
storage) a crafted chain of chunks nested in each other read and held the
square of the file's size (a 192 KB file read 768 MB).

Chunks are now read from a FIFO queue of (address, length) pairs in the
order their continuation messages are found, as H5O_protect does and as
the editor's header walker already did, each buffer released before the
next read. In both header versions a chunk starting at an address seen
before (cycle) is refused, and so are chunks adding up to more than the
file, which bounds a header's reads by the file's size. Overlap itself is
allowed: libhdf5 reads cve-2025-7067.h5, whose continuation chunk overlaps
chunk 0 (refusing overlap cost that conformance file).

Tests: the nested chain is refused having read at most the file (it read
n^2 bytes before); a 3000-chunk chain reads each chunk once; a chunk's
messages follow the whole previous chunk (they were inserted at the
continuation message); an overlapping continuation chunk is read.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 18:50:23 -05:00
osobhandClaude Opus 5.5 930921e8cb edit: shrink by visiting the chunks that exist 
prune_plan stored one Vec<u64> for every chunk coordinate of the region a
shrink cuts off, existing or not, so a sparse dataset exhausted memory
(about 62 bytes per coordinate; (4, 2e7) with chunks (1, 1) took 2.5 GB,
larger extents never finished). It now places each existing chunk in
H5D__chunk_prune_by_extent's walk (its pass, then its coordinates) and
sorts, which gives the same chunks, order and actions in memory and time
proportional to the chunks that exist.

A unit test checks the plan against the full walk (kept as the test's
reference) for 3000 random extents and chunk subsets. The interop test
shrinks a (4, 10^12) dataset with chunks (1, 1) and 9 chunks (v1 and v2
B-tree): 0.56 s and 43 MB peak; the old code aborted on allocation under an
8 GB limit.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 18:50:23 -05:00
osobhandClaude Opus 5.5 159e588550 filters: Fletcher-32 as libhdf5 computes it 
Our checksum reduced its sums with `% 65535`; libhdf5's
H5_checksum_fletcher32 folds them with `(s & 0xffff) + (s >> 16)`, which
leaves 0xffff where the modulo leaves 0. On about one chunk in 32768
libhdf5 refused the chunks we wrote and we refused the chunks it wrote.
Every release since v2.1.0 is affected.

clawhdf5_format::checksum::fletcher32 is a port of H5_checksum_fletcher32
and the filter's only implementation. Verification also accepts the
byte-swapped form libhdf5 accepts (1.6.2 and earlier) and the `% 65535`
form earlier releases wrote, so their files stay readable.

The new interop test compares the checksum with libhdf5's own function
(ctypes) on every 1- and 2-byte input and 40 000 random and fold-heavy
inputs, and moves fold-case chunks between h5py and FileBuilder/FileEditor
in both directions; with the old filters.rs the three file tests fail.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 18:50:23 -05:00
osobhandClaude Opus 5.5 6185874f9c format, clawhdf5: cut every Storage read to the range asked for 
ExtentBytes and read_exact_at/read_upto rejected short results but passed
longer-than-asked ones through, and FileData forwarded them too, so a
Storage that broke read_at's contract by returning extra bytes had them
decoded or returned as data (a contiguous dataset read gained 37 junk
bytes). gather_storage alone trimmed.

- storage::exact_len (new, pub): a read of len bytes as exactly len — cut
  when longer, an error when short. read_exact_at, read_upto and
  ExtentBytes (so chunk fetches and selection gathers) go through it.
- FileData cuts a backend's answer to what it asked for before laying the
  cache image over it.
- Tests: over a storage that appends 37 junk bytes to every read, every
  format-crate fixture reads exactly as from the slice
  (overlong_reads_are_cut_to_the_range_asked_for), and every facade
  fixture opens and reads through File::open_storage as through File::open
  (overlong_storage_reads_identically). Both failed before.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 18:43:05 -05:00
osobhandClaude Opus 5.5 89e7977943 clawhdf5: parse local files through the slice, and cache the contiguous view 
Since M2 the facade handed the metadata parsers its FileData view, so a
local file ran the parsers monomorphised for FileData, whose Storage impl
worked out contiguous() (patched and overlay checks, two range conversions)
on every structure read. A metadata walk of h5stat_newgrat.h5 (35,001
groups: open, entries and attrs of each) was about 4.5% slower than at
8f59b2e.

- FileData works out its contiguous slice once at open (a borrow of its own
  heap/mapped buffer, kept as a pointer; see the SAFETY notes).
- with_bytes! hands the in-memory slice to the format parsers when the file
  has one (header parsing, attributes, group listings and lookups, path
  resolution, shared messages, VL decoding), so local files run the [u8]
  parsers as before; storage-backed files still get FileData.

Provisional A/B on tank (load 5-11), best of 30, 5 alternating rounds:
walk 26.07-26.37 ms at 8f59b2e, 27.24-28.03 ms before this commit,
26.52-26.98 ms after. Caching alone did not move it (27.02-27.47 ms); the
dispatch did. File::open read_f32 on 32M f32 (contiguous, chunked, gzip)
stays within noise of 8f59b2e.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 18:41:10 -05:00
osobhandClaude Opus 5.5 67e72b30d7 clawhdf5-remote: clippy clean with every feature set 
checked_div in the test server's throttle, a slice for the single range
of fetch_first, and dead-code allowances for the redaction helpers in a
build with neither http nor a cloud store.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 18:37:47 -05:00
osobhandClaude Opus 5.5 0e98ffc498 docs: remote files after the adversarial review 
CHANGELOG, the clawhdf5-remote and h5rs READMEs and the remote-files
known issues: redirect rules, scaled timeouts (min_speed), URL redaction,
claimed lengths never allocated (download, --max-download), a 200 for a
small file accepted, and ObjectStoreStorage from any thread.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 18:37:05 -05:00
osobhandClaude Opus 5.5 efb88f94e3 h5rs: check URL opens the remote file once 
open_arg_whole opened and parsed the remote file through open_arg, then
opened it again to download it, so every `h5rs check URL` probed the
server twice. It now opens the storage once and downloads through the
same block cache (whose first block the probe already filled).

Test: check --data of a file within one block costs exactly one request
(two before).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 18:36:35 -05:00
osobhandClaude Opus 5.5 ef480746da clawhdf5-remote: a failed first fetch is Error::Remote 
cached() mapped an error of its open-time prefetch (a network error, a
changed file) to Error::Hdf5(Format(Storage)), misclassifying it for
callers that match on the variant. It is now Error::Remote
(RemoteError::Backend with the backend's message). open_object and the
s3/gs/az URLs fetch the first block of an ObjectStoreStorage directly, so
their errors keep their kind (FileChanged, ObjectStore).

Test: cached() over a backend whose reads fail gives Error::Remote(Backend)
(Error::Hdf5 before).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 18:36:35 -05:00
osobhandClaude Opus 5.5 c5b2afbc35 clawhdf5-remote: ObjectStoreStorage works from any thread 
It refused whenever Handle::try_current() was Ok, which is also the case
inside spawn_blocking threads — so the workaround its own error message
recommended failed the same way, and the backend could only be used from
a bare std::thread in a tokio application.

Reads are now spawned on the storage's own runtime and the caller waits on
a channel: the future never runs on the caller's thread, so neither a
spawn_blocking thread nor a current-thread runtime can deadlock or panic
(a read inside a runtime blocks that thread, like any blocking call; the
docs still recommend spawn_blocking there).

Tests: a read in spawn_blocking of a multi-thread runtime and a read inside
a current-thread runtime's task give File::open's values (both errors
before).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 18:35:36 -05:00
osobhandClaude Opus 5.5 b086dc3c2b format: read a contiguous selection's runs merged across small gaps 
gather_storage merged only runs that touch, so a strided selection of a
contiguous dataset over a Storage became one range (and one owned Vec) per
element: a stride-2 read of 32M f32 through File::open_storage made
16,777,232 read_at calls, took 2.0 s and peaked at 2.09 GB.

The selection is now walked twice. The first walk checks the runs and plans
spans: runs in increasing order at most 4 KiB apart (GATHER_GAP_BYTES) are
read as one span up to 8 MiB (GATHER_SPAN_BYTES; a longer run is split), so
nothing is stored per run. The spans are fetched in RAW_BATCH_BYTES batches
while the second walk copies each run out of its span. Same checks and
errors as before.

The same read is now 32 reads and 0.31 s (File::open: 0.08 s).
contiguous_read_interop: every h5py-checked selection is also read through
File::open_storage and must give libhdf5's bytes; a new test bounds the
range reads of strided, blocked, column and point selections (stride 2: at
most 1 data read; 563,200 before).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 18:34:31 -05:00
osobhandClaude Opus 5.5 30a1ed6b9c clawhdf5-remote: request timeouts scale with the body 
timeout_global (60 s) covered a whole request, and a request can carry
8 MiB (max_request): below about 140 KB/s every block run timed out, was
retried from scratch and failed, so a slow link could not read remote
files at all.

HttpOptions::timeout (now 30 s) bounds connecting and receiving the
response headers; the body gets timeout + its size at the new
HttpOptions::min_speed (16 KiB/s by default: 94 s for a 1 MiB block).
A slow but moving link is not cut off; a stalled one still fails.
(ureq has no idle timeout; its body timeout is a total budget.)

The test server can throttle bodies and stall mid-body. Test: a 256 KiB
block at 256 KiB/s reads with a 300 ms timeout (it failed before), and a
body stalled for 20 s fails in under 5 s.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 18:34:31 -05:00
osobhandClaude Opus 5.5 61e34927dc clawhdf5-remote: a 200 covering the requested range is the whole file 
The first request asks for bytes=0-1048575. RFC 9110 lets a server answer
200 when the range covers the whole representation, so a file under
1 MiB on a server that does support ranges could be refused as 'does not
support range requests'. A 200 whose Content-Length (or, without one, its
body, read at most that far) is within the range asked for is now kept as
the whole file and read from memory; a longer one is still refused unless
allow_full_download is set.

Test: a 9968-byte file served with 200 opens in one request with the
transcript of File::open (it was refused before); with a 4096-byte first
request it is still refused.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 18:32:15 -05:00
osobhandClaude Opus 5.5 680c90b3a8 clawhdf5-remote: redirects are followed safely 
ureq's defaults followed up to 10 redirects, including from https to plain
http, and forwarded the custom HttpOptions::headers (X-Api-Key, Cookie,
...) to whatever host a redirect named — only Authorization was stripped.

HttpStorage now follows redirects itself (ureq's max_redirects is 0):
- at most HttpOptions::max_redirects per request (default 5; 0 refuses
  any redirect), then RemoteError::Redirect;
- never from https to another scheme, nor to a non-http(s) URL;
- once a redirect leaves the URL's origin (scheme, host, port), none of
  the custom headers is sent any more (Authorization included);
- each hop counts as a request; errors show the target redacted.

Tests: a redirect to another local port reads the right data and the
target never sees X-Api-Key or Authorization (it did before); a
same-origin redirect keeps them; a loop stops after 6 requests; 0 refuses;
unit tests for target resolution, the https downgrade and origins.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 18:31:39 -05:00
osobhandClaude Opus 5.5 7d629f49e3 format: bound and batch every chunk fetch over Storage 
Only the full read split its chunk fetches into 64 MiB batches. The
selection path, the indexed read and the parallel_read decoders fetched
every chunk's stored bytes in one read_ranges call, each extent bounded only
by the file length, so a crafted chunk index pointing many chunks at one
large extent made File::open_storage hold chunks x extent bytes (3.3 GB from
a 16.8 MB file) before the first decode error.

- storage::for_each_extent_batch is now the one way raw-data reads fetch
  chunk bytes: batches of at most RAW_BATCH_BYTES (now pub), each decoded
  before the next is fetched. Used by the full, cached, indexed, selection
  and parallel_read paths; the sweep read uses read_extent per chunk.
- ExtentReq carries each chunk's claimed extent (bounds-checked as before,
  same errors) and the prefix actually fetched:
  filters::stored_chunk_limit — the chunk size if unfiltered, else each
  applied filter's worst-case growth (n + n/4 + 4096 per codec; unbounded
  only for an application-registered codec). The in-memory path cuts the
  slice it decodes the same way, so both paths still agree.
- tests/raw_fetch_bounds.rs: a crafted chunked_large.h5 (ten chunks all
  claiming 20 MiB at one padding blob) read through every path over a
  storage that records the largest single fetch; and 160 MiB of legitimate
  unfiltered chunks fetched batch by batch. Before: one 80 MiB fetch
  (selection) and one 160 MiB fetch; after: within the budget.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 18:30:30 -05:00
osobhandClaude Opus 5.5 c04e34620e clawhdf5-remote, h5rs: URLs' credentials are never shown 
Every RemoteError message and HttpStorage's Debug output held the URL as
given, with any user:password@ and the query string — for a presigned
S3/GCS/Azure URL, its signature or token. An application logging the
error leaked the credential.

- New clawhdf5_remote::redact_url: no userinfo, no fragment, query values
  replaced by REDACTED (plain key names kept).
- HttpStorage formats every message with the redacted URL, and scrubs the
  URL's secret parts from errors of the HTTP client (whose texts can echo
  the URI); Debug shows the redacted URL. storage_for_url's and the object
  store URL errors are redacted too. HttpStorage::url() still returns the
  URL as given, documented as not for logging.
- h5rs prints FILE arguments that are URLs redacted: in errors and in
  dump/stat/check/diff output.
- The test server can force a status and send a wrong Content-Range.

Tests: 404, 403 (at open and on a read), wrong Content-Range (at open and
on a read), no range support, encoded body, ETag change, timeout,
connection closed and bad scheme errors, Display and Debug, contain none
of the secrets; h5rs likewise for every subcommand.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 18:30:02 -05:00
osobhandClaude Opus 5.5 8df5b209a7 clawhdf5-remote, h5rs: never allocate a length the server only claims 
h5rs check URL read the whole file with one read_at(0, len), len being
whatever Content-Range said. BlockCache listed every block index of the
span and preallocated len bytes: a server claiming 2^62 bytes for a 10 KB
file made h5rs abort (memory allocation of 35184372088832 bytes failed).

- BlockCache: a read spanning more than the budget (or eight max_requests)
  is fetched piece by piece and not kept, its output growing only as
  data arrives; read_ranges falls back to that per range; prefetch is
  clamped to the budget.
- New clawhdf5_remote::download(storage, max_bytes): refuses a claimed
  length above the limit (RemoteError::TooLarge) before any request, then
  reads in 64 MiB steps. New RemoteError::Backend for read errors.
- h5rs check downloads through it, with --max-download N (default 1 GiB).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 18:27:14 -05:00
osobhandClaude Opus 5.5 e8aaf050be clawhdf5-remote: no overflow on lengths near u64::MAX 
A server can claim any length in Content-Range. block_len computed
start + block_size, which overflowed in the last blocks of a file claimed
to be near u64::MAX (a panic in debug builds, a wrapped value in
release); insert() multiplied block indices unchecked. The cache's block
arithmetic is now saturating/checked, and a run that does not split into
whole blocks is an error instead of an endless loop or a slice panic.

The test server gains fake_total (claim a length, serve zeros past the
data); a test reads the last bytes of such files and opens a file whose
superblock EOF and root addresses sit near u64::MAX.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 18:24:48 -05:00
osobhandClaude Opus 5.5 5062b907bd clawhdf5-remote tests: the server counts only requests for its files 
A local port scanner's GET / reached the test listeners and was counted,
failing the exact request budgets (and consuming injected 503s). Requests
for paths the server does not serve are now answered 404 without being
counted, delayed or failed; the query string is not part of the path.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 18:23:27 -05:00
osobhandClaude Opus 5.5 4f5697fdd9 clawhdf5-remote: readers waiting on a failed fetch get its error 
A reader that waited for another reader's fetch of a block got "the
fetch of this block failed" when that fetch failed, not why: a file
replaced on the server while open was reported as FileChanged to one
thread and as an anonymous failure to the others. The fetch's error is
now handed to every reader waiting on it.

Regression test: four threads read the same block from a slow backend
whose fetches fail with a "changed while open" error; each gets that
error (it failed for the waiters before this change).

Also fixes the ignore-Range test, broken by the previous commit: the
test server now counts a body before sending it, so "the refused body
was not read" is checked as "refused at the first response".

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 17:31:26 -05:00
osobhandClaude Opus 5.5 955dd1c691 docs: range-read milestone M3 — remote files 
README: "Reading remote files" (open_url, the range_server and read_url
examples with their real output against the fixtures, h5rs on URLs), the
crate in the crate map and the unreleased highlights. CHANGELOG: the
clawhdf5-remote crate, h5rs URLs, File::storage and
VlResolver::element_in, with the request counts over the conformance
corpus (tank, 2026-09-26, the command given). known-issues: the M2
range-read entry updated (the cache now exists; h5rs reads through
storage) and a new entry for the remote backends' limits (no Python or
browser URLs yet, fixed block size, cloud stores not run against a real
bucket, validators, credentials). Design doc: M3 status with the choices
that differ from the plan (a crate rather than a clawhdf5-io feature,
ureq for HTTP so the default build has no C) and the corpus counts.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 17:29:09 -05:00
osobhandClaude Opus 5.5 ebe51f8e97 clawhdf5-remote tests: open + list and a dataset read counted apart 
The per-file report now separates a tree view (open, every group's
entries, every dataset's shape and type) from reading the largest
dataset under 64 MiB, and checks the budget the design's testing section
asks for: listing the IMERG file (file A of docs/design/range-reads.md
section 2) takes at most 3 requests when CLAWHDF5_REMOTE_CORPUS includes
it. The test server now counts a response's bytes before sending it: a
client could read a body and reset the counters before the server thread
had added it, so the counts of the next file were occasionally too high.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 17:26:57 -05:00
osobhandClaude Opus 5.5 4e8109770d tests: shrink and regrow datasets allocated early 
Datasets with early allocation and unlimited dimensions (Extensible
Array, version-2 B-tree, version-1 B-tree under earliest), unfiltered
and deflated: the random resize workload gives the values h5py gets and
the same chunk index shape, with every chunk a growth brings in
allocated and filled as H5D__chunk_allocate does.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 17:25:28 -05:00
osobhandClaude Opus 5.5 c513f7e6d7 h5rs: URLs as FILE arguments (feature remote) 
With the `remote` feature (`remote-https` for https://), ls, dump, stat
and diff take an http(s):// (or s3://, gs://, az:// with those
clawhdf5-remote features) URL wherever they take a file, and read it by
range requests through clawhdf5-remote's block cache. check validates
every byte, so it downloads a remote file whole and checks it as before.
Without the feature a URL is a clean error naming it.

The tools read the file through File::storage instead of as_bytes: object
headers, shared messages, attributes, v1 and v2 group links, dense
storage (fractal heaps and v2 B-trees), path resolution, chunk listings
and variable-length values go through the format crate's *_in functions,
and the fractal-heap block verifier reads each block through the storage
(a read failure of a remote file is reported as a problem, not as "past
the end of the file"). A local file's storage is its mapped bytes, so its
reads are still slices. stat's file size comes from the opened file, so
it is right for a URL.

Tests: tests/remote.rs serves fixtures (old and new formats, a paged
file, a metadata cache image, a multi-block fractal heap, compounds, v1
groups) with the clawhdf5-remote test server and requires every
subcommand's output and exit status for the URL to equal the local
file's, and diff of the two to be clean; 404s, non-HDF5 bodies and
https without its feature are clean errors. Local output is unchanged:
the old and new h5rs print the same for ls -r -v, dump, stat and check
--data on the 747 conformance and CVE corpus files (tank, 2026-09-26;
the dumps of h5diff_hyper1/2.h5 were too large for the comparison
script, their ls, stat and check agree), except cve-2025-2310.h5, whose
dump error messages differ between runs of the old binary too (which
failing chunk is reported first).

ci-test.sh lints h5rs with remote-https, runs the URL tests and checks
h5rs with remote for C.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 17:23:53 -05:00
osobhandClaude Opus 5.5 a4f586e657 format: VlResolver::element_in and string_element_in over any Storage 
VlResolver::element and string_element return slices of the whole file,
so they exist only for a resolver over &[u8]. Their *_in forms work for
any Storage (a remote file): the element's bytes borrowed from the
resolver's cache of heap collections, with the same null-element, NUL
and size checks. h5rs decodes variable-length values with them.

Test: over a read_at-only storage they give what element/string_element
give over the slice, for a string with an embedded NUL, a null element
and an element whose heap object has the wrong size.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 17:20:50 -05:00
osobhandClaude Opus 5.5 c54c64cc9b clawhdf5: File::storage gives the file's view as a Storage 
The bytes File::as_bytes returns (from the superblock on, bounded by the
recorded end of file, a metadata cache image laid over), as a
&(dyn Storage + Send + Sync) for every backend. Code that parses the file
itself with the clawhdf5_format *_in functions — h5rs does — can then
read a file opened with File::open_storage (a remote file) as well as a
local one; in memory its as_contiguous() is as_bytes(), so local reads
stay slices.

Test: for every fixture, File::open's storage() is as_bytes() as its
contiguous view, and File::open_storage over a read_at-only storage gives
the same bytes through storage().read_at.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 17:16:35 -05:00
osobhandClaude Opus 5.5 4ff3e40fea clawhdf5-remote: object stores through object_store (S3, GCS, Azure) 
ObjectStoreStorage (feature `object-store`, pure Rust) reads one object
of any object_store store by ranged get_opts, pinned at open by a head
request: If-Match with its ETag (and the ETag and size of every response
compared), else its version or modification time. A change is
RemoteError::FileChanged. object_store is async and Storage is not, so
the storage owns a small multi-threaded tokio runtime (two workers) and
blocks the calling thread on it; the ranges of one read_ranges call are
fetched concurrently (up to 8). From inside another tokio runtime it
refuses with RemoteError::Usage instead of blocking a worker, and it
shuts its runtime down in the background on drop so dropping it in async
code does not panic.

open_object(store, path, options) opens a file through a block cache
(first block prefetched); open_url accepts s3://, gs:// and az:// with
the `s3`, `gcs` and `azure` features, configured from the environment by
object_store's from_env builders. Those pull object_store's cloud clients
and aws-lc-rs (C), so they are opt-in; without them the URL is a clean
UnsupportedScheme error naming the feature.

Tests against object_store's in-memory and local-file stores (no cloud):
every fixture's transcript equals File::open's, a multi-block object is
fetched in coalesced block runs, an object replaced while open is an
error, and a missing object or a read from inside a runtime is a clean
error. ci-test.sh lints all backends, runs these tests (with s3 for its
URL parsing test) and checks object-store for C in the no-C step.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 17:15:06 -05:00
osobhandClaude Opus 5.5 0aca0eb724 docs: editor coverage — version-2 B-trees, shrinking, dense attributes, reuse 
CHANGELOG (Unreleased): the new FileEditor operations, space reuse, and
the two reader fixes (implicit index grid, object-header continuation
chains). known-issues: the editor's remaining refusals (skipped heap
blocks, heaps with filters or child indirect blocks, freeing a heap
block, implicit-index insertions, ...) and the append-waste sizes before
and after reuse (measure_append_waste, tank 2026-09-26; file sizes are
deterministic). range-reads design: status note on the reader changes.
README and CLAUDE.md: what the editor covers and how to test it.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 17:13:31 -05:00
osobhandClaude Opus 5.5 db2554dd81 clawhdf5-remote: block cache and HTTP range reads (open_url) 
Range-read milestone M3, first half: a new crate with the block cache the
design makes mandatory for remote files and an HTTP backend, so
open_url("http://...") gives a clawhdf5::File over File::open_storage.

BlockCache wraps any Storage: aligned blocks (1 MiB by default, the size
docs/design/range-reads.md section 2 measured), LRU with a byte budget,
the missing blocks of one read_at/read_ranges fetched with one backend
read_ranges call as runs of consecutive blocks (a one-block gap filled to
merge runs, each request at most 8 MiB), and reads that miss more than
half the budget not kept. Thread-safe without holding the lock across a
fetch: a block being fetched is in flight, a second reader waits for it
instead of fetching it again, and a failed fetch fails its waiters and is
not cached. A backend holding the file in memory passes through.

HttpStorage (ureq, no TLS by default; `https` adds rustls with ring):
opening is one ranged GET of the first block, whose Content-Range gives
the length (the cache keeps the bytes). The file is pinned by a strong
ETag (If-Match), else Last-Modified (If-Unmodified-Since), and its length,
checked on every response: a change is RemoteError::FileChanged, never
mixed data. A server that ignores Range is refused without reading the
body unless a full download is allowed. Connection errors, timeouts,
408/429/5xx and short bodies are retried with exponential backoff;
Accept-Encoding: identity, and an encoded body is refused. read_ranges
fetches its ranges in parallel.

Tests (a std-only HTTP/1.1 server in tests/common/server.rs, also the
range_server example): every fixture read over HTTP gives File::open's
transcript (CLAWHDF5_REMOTE_CORPUS adds the conformance corpus), with
request counts per file with and without the cache; an h5py-written file
against libhdf5's values; a multi-block file fetched in whole blocks, each
once; a server ignoring Range; a file replaced mid-read (ETag,
Last-Modified, length only); truncated bodies and 503s (retried, then an
error, never cached); a slow server with 8 concurrent readers (no block
fetched twice); bad URLs, 404, encoded bodies, non-HDF5 data. The cache
has unit tests for coalescing, splitting, LRU order, large reads,
failures and concurrent in-flight dedup.

ci-test.sh: clawhdf5-remote joins the no-C default-build check, and its
https feature is linted.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 17:13:28 -05:00
osobhandClaude Opus 5.5 955fdb660d tests: resize and write clawhdf5-written datasets on every chunk index 
Version-2 B-tree (its writer's own node size and single-leaf layout),
Extensible Array and Fixed Array datasets written by FileBuilder, with
and without deflate, resized up and down along both dimensions and
written at random against a model; h5py, h5dump and h5rs check read the
result, and h5py resizes and rewrites every dataset afterwards.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 17:12:14 -05:00
osobhandClaude Opus 5.5 304aed5813 tests: random editor operations include shrinking, 2-D growth, dense attributes 
random_operations_match_a_model (edit_interop) now drives, on every
libver (earliest, v114, latest) and filter set (none, gzip + shuffle +
fletcher32, LZF):

- a dataset with one unlimited dimension and one with two (a version-2
  B-tree chunk index under v114/latest), resized to random shapes that
  shrink and grow any resizable dimension, with block and point writes;
- attributes on the first dataset under 20 names with values of random
  types and sizes (scalars, int64 arrays, short strings, strings above
  the heap's managed limit), so they move to dense storage on version-2
  headers and are replaced by values of other sizes;

against a model where shrunk-away elements that come back read as the
fill value, compared with our reader and with h5py/numpy every 40 steps,
with h5dump and h5rs check; h5py then grows both datasets and adds an
attribute. The attribute check also compares h5py's attribute count with
libhdf5's object info. CLAWHDF5_EDIT_SEED reruns the workloads with other
random choices (seeds 1000-4000 pass).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 17:11:44 -05:00
osobhandClaude Opus 5.5 1ffd013de9 edit: delete a heap's huge-object B-tree with its last huge object 
libhdf5 deletes a fractal heap's huge-object B-tree when the heap is
closed with no huge object left (H5HF__huge_term) and starts huge IDs
over. The editor left the empty tree, and a read-only libhdf5 then
failed to list the object's attributes: closing the heap tried to delete
the tree ("no write intent on file"), and h5dump failed the same way.

Replacing an object's last huge attribute (one above the heap's 4 KiB
managed limit) now deletes the tree (header and nodes freed), resets the
next huge ID and the wrapped flag, as libhdf5 does; a later huge
attribute creates a new tree. Bt2::delete frees a whole tree.

Found by the extended random-operation test; regression:
last_huge_attribute_replaced (fails before: h5dump, h5py listing), which
also compares the heap with libhdf5's after the same replacement.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 17:11:36 -05:00
osobhandClaude Opus 5.5 dc9cfba6bb edit: reuse space freed earlier in the editing session 
A FileEditor now keeps the space its edits free — a filtered chunk that
moved, chunks a shrink removed, B-tree nodes merged away, a heap's
replaced root indirect block or free-space section info, huge objects
replaced — and later edits allocate from it (best fit, lowest address
among equals, zeroed) before growing the file. An edit never reuses what
it frees itself: until it is committed the file still refers to that
space. Reused blocks are written in the commit's first phase with the
space past the old end of file (nothing on disk refers to them yet),
before any existing byte changes, so the crash-safety ordering holds.
Space still free when the editor is dropped is leaked, as libhdf5 leaks
it without a persistent free-space manager (files that have one, or use
paged aggregation, are still refused at open).
FileEditor::reusable_bytes reports what is left to reuse.

Tests: FreeList merging and best fit, the edit-local rule and the commit
split (image unit tests); a shrink followed by regrowth writing the same
data reuses every removed chunk and leaves the file size unchanged,
while one editor per edit grows the file, h5py/h5dump/h5rs check read
both and h5py continues (freed_space_is_reused_within_a_session).

measure_append_waste (edit_interop, ignored), same workload, one editor,
before -> after (bytes; libhdf5 in brackets), on tank 2026-09-26:
gzip chunks 1024, 1000 appends of 100: 307210 -> 306780 (306058);
gzip chunks 4096, 2000 appends of 10: 119684 -> 79829 (50292);
unfiltered unchanged (no space is freed).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 17:04:15 -05:00
osobhandClaude Opus 5.5 773f427f16 edit: dense attributes, compact-to-dense transition, creation order 
FileEditor::set_attr now handles every attribute storage libhdf5 uses
for version-2 object headers:

- objects that track (and index) attribute creation order: compact
  attributes carry their creation index in the message header, the
  Attribute Info message its maximum;
- the move to dense storage when an object reaches its compact limit (or
  an attribute is too large for a header message), as H5O__attr_create
  does it: a new fractal heap, name index (v2 B-tree type 8) and, when
  creation order is indexed, creation-order index (type 9); the compact
  attributes moved over in header message order, their messages freed;
- objects already in dense storage (h5py- or clawhdf5-written): new
  attributes inserted (H5A__dense_insert), an attribute replaced by one of
  the same encoded size rewritten in its heap object (H5A__dense_write),
  otherwise removed from both indexes and the heap and inserted anew.

edit/fheap.rs follows H5HF: managed objects go to the best-fitting free
section of the heap's free-space manager (FSHD/FSSE, kept as libhdf5
keeps it — sorted sections, counts, section info reallocated when its
size changes, the manager deleted when empty); otherwise to a new direct
block: the root direct block of an empty heap, else the block at the
allocation iterator in the root indirect block (created from the root
direct block, doubled as needed), with libhdf5's managed-space, allocated
space, free space and iterator bookkeeping. Objects above the managed
limit are huge objects in their own space, indexed by the huge-object
B-tree (type 1). Removed objects return their space merged with adjacent
free space. Refused before anything is written: heaps with I/O filters,
child indirect blocks, an object larger than the next heap block
(libhdf5 skips blocks and records them as free space), free sections
other than those inside direct blocks, removing a direct block's last
object (libhdf5 frees the block), directly addressed huge objects.

Attributes are encoded as libhdf5 does when h5py opens a file r+ (low
bound "earliest"): message version 1 (3 for non-ASCII names), simple
dataspaces with their maximum dimensions. Header chunks are now visited
in libhdf5's order (FIFO), which is also the order attributes move to
dense storage in.

Tests (edit_coverage_interop): 40 attributes on each of a plain group, a
group tracking and indexing creation order, and a dataset (earliest,
v110, latest), some above the 4 KiB managed limit, then same-size
rewrites: the heap statistics, free-space sections and both index
B-trees node for node equal libhdf5's doing the same through h5py; then
replacements of other sizes, h5py adds/deletes/rewrites; h5py, h5dump,
h5rs check and our reader agree throughout, h5py's attribute count
included. clawhdf5-written dense storage (tracked and untracked) is
extended the same way; refusals leave the file byte for byte as it was.
edit_interop's attribute test now expects dense storage and tracked
creation order to work.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 16:58:08 -05:00
osobhandClaude Opus 5.5 7e5e920c72 format: read object headers with long continuation chains 
libhdf5 follows any number of continuation chunks, and a header that is
full gains one per message added (each new chunk holding the next
continuation message), so a version-1 header with a few dozen attributes
added one at a time is a chain dozens of chunks long. The reader recursed
once per chunk and refused a chain deeper than 32 (NestingDepthExceeded):
h5py read such files, we did not. Version-2 headers stopped at 256
continuation chunks.

Version-1 chunks are now followed with an explicit stack (the same
depth-first message order as before), version-2 ones as before; both
refuse a chunk address seen twice (a cycle, what the limits guarded
against) and more than 65 536 chunks.

Regression: long_v1_continuation_chains_are_read (a 200-chunk chain),
v1_continuation_cycles_are_refused; the dense-attribute interop test's
'earliest' case produces such a chain.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 16:56:09 -05:00
osobhandClaude Opus 5.5 f191dc09d5 docs: range-read milestone M2 — changelog, limits, design status 
CHANGELOG (Unreleased): File::open_storage, raw data and v2 B-trees over
Storage, the tests and their corpus results (2026-09-26, tank; conformance
600 of 697, results.json identical to 8f59b2e). known-issues: what
open_storage does not do yet (no remote backend or block cache, read_at
counts of a one-pass read, v1 group lookups, whole-file VDS sources,
zero-copy methods, SWMR growth, hash-order error choice on damaged chunked
datasets). Design: M2 status and the choices made.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 16:49:42 -05:00
osobhandClaude Opus 5.5 1c3ef98828 clawhdf5: File::open_storage reads any Storage through the full API 
File::open_storage(Arc<dyn Storage + Send + Sync>) opens a file served by
any backend: groups, datasets, attributes, read_*, selections, VL data
and virtual datasets (external sources through the new
File::set_vds_resolver) all read through Storage::read_at/read_ranges.
The file's view (user block skipped, bounded by the recorded end of
file) is itself a Storage; File::open and File::from_bytes keep their
mmap and in-memory paths, now as that view's as_contiguous() fast path.
A storage-backed file's metadata cache image is laid over each read it
covers (new CacheImage::entries), as libhdf5 loads it.

The typed readers keep their fast paths over any storage: a contiguous
dataset is read in one piece and converted (read_f64 and friends), and a
contiguous native selection reads only its runs
(data_read::read_selection_native_in). Zero-copy methods answer
ContiguousStorageRequired when the bytes are not in memory, and
File::as_bytes panics there (File::contiguous_bytes is the fallible
form).

Test: tests/storage_equivalence.rs reads every fixture, and with
CLAWHDF5_STORAGE_CORPUS every conformance-corpus file, through File::open
and through open_storage over a read_at-only CountingStorage — tree,
attributes, and every dataset's values several ways — and requires
identical transcripts; it prints the read_at calls and bytes a one-pass
read costs per file.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 16:41:47 -05:00
osobhandClaude Opus 5.5 e9c71e5d2e edit: version-2 B-tree chunk indexes, shrinking, early allocation 
FileEditor can now:

- add, move and resize chunks of datasets with two or more unlimited
  dimensions (version-2 B-tree chunk index, record types 10/11). The new
  edit/btree2.rs follows libhdf5's H5B2 code: H5B2_update (modify, or
  insert into a leaf with room, or fall back to H5B2__insert), the
  preemptive split/redistribute loop with its two retries, split1,
  split_root (depth growth, node geometry per depth), redistribute2/3,
  cumulative record counts and the pointer widths H5B2__hdr_init derives,
  a checksum per node. Removal (H5B2_remove: merge2/3, redistribution,
  root collapse, the internal-record swap with a leaf) is there too. A
  dataset without an index yet gets one from the layout message's node
  size and split/merge percentages.
- shrink a chunked dataset along any dimension (resize to a smaller
  shape), as H5D__set_extent / H5D__chunk_prune_by_extent do: the same
  chunks visited in the same order; chunks wholly outside the new extent
  are removed from the index (version-1 B-tree: H5B_remove with its
  sibling key and link fix-ups and the empty-root case; version-2
  B-tree; Fixed/Extensible Array elements reset to the fill element;
  an implicit index keeps its chunks, as libhdf5 does) and their space
  noted as free; the part of each partial edge chunk outside the extent
  is overwritten with the fill value, so elements that come back after
  a later growth read as fill.
- under early allocation, allocate and fill the chunks a growth brings in
  (H5D__chunk_allocate), which an implicit index needs: libhdf5 refills
  them, and they may hold the data of chunks pruned earlier.

Tests (crates/clawhdf5-tools/tests/edit_coverage_interop.rs): growth in
both dimensions of v110/latest files, unfiltered and deflated, gives
node-for-node the version-2 B-tree libhdf5 builds (h5py with its chunk
cache off, so chunks enter the index in the editor's order), through a
depth increase; random chunk order; 60 random shrink/grow/write steps on
Extensible Array, version-2 B-tree, Fixed Array, 1-D and implicit
datasets (earliest/v110/latest, with and without gzip+shuffle) give the
values h5py gets doing the same and the same index shape (version-1 and
version-2 B-tree node shapes, Extensible Array statistics); h5py r+
continues on every result; h5dump and h5rs check accept them.
edit_interop's version-2 B-tree case now appends instead of expecting a
refusal; shrinking is no longer an error in edit_tests.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 16:35:26 -05:00
osobhandClaude Opus 5.5 17201e279d format: implicit chunk index addresses over the maximum chunk grid 
libhdf5 allocates an implicit index's chunks for the whole maximum
extent and places chunk `scaled` at its row-major position in the
maximum chunk grid (H5D__none_idx_get_addr, max_down_chunks). The reader
used the current grid, so a dataset below its maximum shape with more
chunk columns at its maximum read other chunks' values from the second
chunk row on (h5py early allocation, fixed maxshape).

generate_implicit_chunks_in_grid takes the maximum dimensions;
generate_implicit_chunks keeps its signature (grid = current extent).
Regression: implicit_chunks_use_the_maximum_grid here, and
implicit_index_below_its_maximum_reads_like_libhdf5 (h5py file) with the
editor tests.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 16:35:06 -05:00
osobhandClaude Opus 5.5 3fa5ed1dda format: raw data, VDS and VL data over Storage 
Every raw-data path has a generic *_in core, with the &[u8] functions as
thin wrappers: data_read (read_raw_data*, read_raw_data_selection,
read_chunked_native), chunked_read (the v1 B-tree chunk index, list_chunks,
the full, cached, sweep and indexed reads), parallel_read, partial_read,
fill_value (read_full_with_fill, apply_to_unallocated_chunks; and
dataset_fill_value_from_storage is now generic), vds (the virtual file
through Storage, external sources still through the resolver),
vl_data (VlResolver<'a, S = [u8]>, read_vl_strings_in, read_vl_bytes_in),
AttributeMessage::read_vl_strings_in and provenance::verify_dataset_in.

With the whole file in memory nothing changes: chunks and contiguous data
are sliced from it as before. Otherwise a chunked read lists its chunks,
fetches their stored bytes with one Storage::read_ranges call per 64 MiB
batch (chunks the cache already holds are not fetched), then decodes as
today; a selection fetches only the chunks it overlaps, and a contiguous
selection only its runs. Each extent's bounds error is the one the slice
code gave, reported when that extent is reached, so errors keep their
order.

Tests: the equivalence harness now reads every dataset's values (whole,
fill-aware, cached, indexed, three selections, VDS, VL strings and
sequences) through the read_at-only storage and requires the slice
results (all 653 corpus files agree); a misbehaving storage (a failing
Nth read, short reads) only ever yields errors or the right values; and
chunked reads are checked to use one read_ranges call.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 16:28:01 -05:00
osobhandClaude Opus 5.5 42894bf93b format: v2 B-trees, dense groups and group listings over Storage 
BTreeV2Header::parse_in, collect_btree_v2_records_in and
find_btree_v2_records_in read one bounded window per node (its size is
known from the parent before the node is read; a count stretched past
node_size is checked against the end of the file first), with the
whole-file bounds errors unchanged. With them, dense attributes, a SOHM
B-tree index and huge fractal-heap objects no longer answer
ContiguousStorageRequired, and group_v1/group_v2 listings, lookups and
path resolution get *_in cores (resolve_group_children_in,
resolve_child_in, resolve_path_any_in, ...). The &[u8] functions are
thin wrappers, as in M1.

The equivalence harness now fails on any ContiguousStorageRequired and
compares v2 B-tree headers, records and descents, group listings, child
lookups and paths; a unit test compares a two-level tree through a
read_at-only storage truncated at every length and with every node byte
flipped.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 16:13:01 -05:00
osobh 8f59b2e1c2 Merge pull request 'Range reads M0/M1 (indexed lookups, Storage trait), ZFP, in-place editing' (#17) from feat/p3-range-zfp-edit into main
CI / test-arm64 (push) Successful in 1m30s
Details
CI / test (push) Successful in 14m27s
Details 
Reviewed-on: #17
 2026-09-26 20:42:21 +00:00
osobhandClaude Opus 5.5 0645dcf173 docs: data-read throughput unchanged by the Storage conversion
CI / test-arm64 (pull_request) Successful in 1m30s
Details
CI / test (pull_request) Successful in 14m22s
Details 
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 15:22:54 -05:00
osobhandClaude Opus 5.5 cadd27df5b docs: ObjectHeader::parse A/B rechecked on an idle machine 
+6-7% (about 4 ns per header) is real; symbol-table nodes -17%, group
B-tree walk -16%, facade listing -2.4%: local metadata reads are net
slightly faster than main.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 15:15:32 -05:00
osobhandClaude Opus 5.5 b49ec39aff docs: conformance report after range-read M0/M1, ZFP and in-place editing (600 of 697 ok) 
Regenerated on tank: ok 599 -> 600 (h5ex_d_zfp.h5), our-error 4 -> 3,
mismatch 2, no panics, hangs, crashes or OOM. Baseline raised.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 15:06:18 -05:00
osobh 8fadb9f424 Merge branch 'feat/p3-in-place-modify' into feat/p3-range-zfp-edit 
# Conflicts:
#	CHANGELOG.md
#	crates/clawhdf5-py/src/lib.rs
#	crates/clawhdf5/src/error.rs
 2026-09-26 14:52:55 -05:00
osobh c233fbca6e Merge branch 'feat/p3-zfp' into feat/p3-range-zfp-edit 
# Conflicts:
#	CHANGELOG.md
#	crates/clawhdf5-format/Cargo.toml
 2026-09-26 14:52:22 -05:00
osobh 437e81cfff Merge branch 'feat/p3-storage-trait' into feat/p3-range-zfp-edit 
# Conflicts:
#	CHANGELOG.md
#	crates/clawhdf5-format/src/attribute.rs
#	crates/clawhdf5-format/src/btree_v1.rs
#	crates/clawhdf5-format/src/data_layout.rs
#	crates/clawhdf5-format/src/extensible_array.rs
#	crates/clawhdf5-format/src/fixed_array.rs
#	crates/clawhdf5-format/src/fractal_heap.rs
#	crates/clawhdf5-format/src/local_heap.rs
#	crates/clawhdf5-format/src/shared_message.rs
 2026-09-26 14:51:51 -05:00
osobh 234dd3e36c Merge branch 'perf/p3-indexed-lookups' into feat/p3-range-zfp-edit 2026-09-26 14:46:14 -05:00
osobhandClaude Opus 5.5 0e8522cfad clawhdf5-format: the writer skips optional filters that fail, as libhdf5 does 
FileBuilder stored every chunk of an LZF or Blosc dataset through the
filter with filter mask 0. libhdf5 counts LZF and Blosc output no smaller
than the chunk as a failure of the optional filter and stores the chunk raw
with the filter's mask bit set. For an LZF chunk whose stream was exactly
the chunk's size, the first libhdf5 rewrite stored raw data at the same
size and kept our stale mask 0 in the index, and h5py could no longer read
the dataset.

precompress_chunks now runs chunks through compress_chunk_masked (as
FileEditor does since f7e2ab1), sequentially and on the parallel path, and
build_chunked_data_from_precompressed records each chunk's real mask in
every index the writer builds: single chunk (layout field), Fixed Array and
Extensible Array filtered elements, and version-2 B-tree type 11 records
(create_datasets_parallel goes through the same path). The writer builds
no version-1 B-tree or implicit index. PrecompressedChunks::chunks gains
the mask. Files whose chunks all compress are byte-identical.

Latent only in the unreleased LZF/Blosc writer (added 2026-09-26); no
tagged release writes either filter.

Tests:
- plugin_filters_interop skipped_optional_filters_are_masked_as_libhdf5_masks_them:
  LZF, shuffle+LZF+fletcher32 and Blosc over random, compressible and
  alternating chunks in every index; masks equal an h5py-written twin's;
  h5py r+ rewrites and extends them; h5py, h5dump and our reader read
  every value. Before: 20 of 24 datasets had masks other than h5py's, and
  with that check disabled h5py failed to read the rewritten datasets
  ("filter returned failure during read").
- plugin_filters_interop files_whose_chunks_all_compress_are_unchanged:
  pins the pre-fix bytes of five all-compressing files.
- chunked_write skipped_lzf_chunks_are_masked_in_every_index (fails before:
  mask 0, want 2).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 14:45:51 -05:00
osobhandClaude Opus 5.5 895c79a2fe docs: M1 changelog after review: probe identity, speed, bounded reads 
- Per-file probe output is identical for 696 of 697 files, not all:
  cve-2025-2310.h5's error string depends on which parallel chunk decode
  fails first, at f2ff2c4 as on this branch.
- The parser cores are generic (S: Storage + ?Sized); provisional A/B
  numbers against f2ff2c4, including the one bench that still shows
  ObjectHeader::parse slower when old and new are separate binaries.
- Reads sized by untrusted fields are bounded; the harness only accepts
  the known whole-file fallbacks.
- range-reads.md records why M1 went generic rather than &dyn.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 14:34:35 -05:00
osobhandClaude Opus 5.5 76c97f6c94 format: bound Storage reads that hostile size fields could stretch 
On a backend without the file in memory, a structure read whose length
comes from untrusted header fields was clamped only by the end of the
file, so a crafted size made one read (and copy) of up to the rest of
the file. Each such read now covers what the parser actually uses:

- local heap names: read in growing pieces (64 bytes first, then 4x)
  up to the end of the data segment, instead of the rest of the segment
  per name (quadratic for a big symbol-table group);
- fractal heap indirect blocks: the doubling-table geometry locates the
  entry covering the object, and the first read ends at that entry; only
  if it is unallocated does the walk read the rest of the block (it
  visits every entry then). One walk implementation serves both;
- paged fixed/extensible array data blocks over 1 MiB: the prefix and
  page bitmap, then each page in use on its own (smaller blocks are
  still one read);
- blocks under one checksum (non-paged array data blocks, extensible
  array index and super blocks): the bounds check that comes first (the
  checksum's; the page bitmap's for a super block) is made against the
  file length before reading (Window::check_extent), so a block claimed
  past the end of the file costs no read. With the checksum feature off
  the parser has no such first check and the old read stands.

Other windows were already bounded (the superblock and object header
prefixes, the fractal heap header by a u16, SOHM tables by u8/u16
counts) or are exact reads checked against the file length first.
In memory nothing changes: the pieces are borrowed slices.

Tests: CountingStorage over a crafted heap (16 MiB file, width and rows
0xFFFF: under 1 KiB read, 16.7 MB before), a heap segment claiming 64 MiB
(one 64-byte read per short name), long names at every piece boundary,
a fixed array block claimed past the end of a 16 MiB file (under 64
bytes read), and in the equivalence harness an h5py file with a 2.4 MB
fixed array block and a >1 MiB extensible array block, whole and cut at
97 points: every chunk index agrees with the slice read and the largest
takes 205 KB (2.4 MB and 1.2 MB when read whole).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 14:31:46 -05:00
osobhandClaude Opus 5.5 e5359354b7 format: equivalence harness only accepts the known whole-file fallbacks 
The harness counted ContiguousStorageRequired from any parser as an
allowed difference, so a converted module that wrongly fell back to the
whole file would still pass. It now accepts the error only from the
three sites that are not converted yet (dense attribute storage, a SOHM
B-tree index, huge fractal-heap objects, all found through a v2 B-tree)
and only in the checks that can reach them; anything else fails with
the check and the site named.

Checked by making LocalHeap::parse_in return the error first: the
fixture and h5py runs fail ("local heap fell back to the whole file").

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 14:22:42 -05:00
osobhandClaude Opus 5.5 052098bf36 format: monomorphise the Storage parsers so local files stay as fast 
Every `*_in` core and the read helpers take `file: &S` with
`S: Storage + ?Sized` instead of `&dyn Storage`, and the `&[u8]`
wrappers pass the slice itself, so they compile to a `[u8]` instance:
`as_contiguous()` inlines to `Some(self)` and each structure read is the
slice code's bounds check again, with no indirect call. `&dyn Storage`
still works (`S = dyn Storage`); there is one parser implementation.

Also, so the structure reads cost no more than the slice checks did:
- ObjectHeader::parse_in reads the prefix once (signature included)
  instead of the signature and then the prefix: two reads for a
  one-chunk header instead of three on a range backend;
- the symbol-table node and group B-tree (v1) loops walk their entries
  with chunks_exact over the bytes read, and the node's redundant second
  bounds check is gone (the entries' read is the check, same error);
- a version-1 header's message list is sized from its (capped) count.
Same results and errors; the unit and equivalence tests are unchanged.

New Criterion bench `clawhdf5/benches/local_metadata_bench.rs` over a
400-group version-1 file written by h5py (new fixture
`v1_groups_400.h5`): ObjectHeader::parse, symbol-table nodes, the group
B-tree walk and a facade listing, using only APIs that exist at f2ff2c4
so it builds there for an A/B.

Provisional A/B against f2ff2c4 (busy machine, not for docs): both
builds linked into one binary and timed in alternation, 200 rounds;
median ratio new/old: facade listing -0.5% to -3.5% (was +14%),
ObjectHeader::parse +1% to +2% (was +25%), symbol-table nodes -18%,
group B-tree walk -18%, local-heap names and resolve_group_children
within +-1.5%. An old-vs-old-copy run shows +-2% from code layout alone.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 14:22:37 -05:00
osobhandClaude Opus 5.5 fe377266e1 clawhdf5: FileEditor unmaps the file before an edit writes it 
Each edit planned over the reader's memory map of the file and committed
while that File, and the Image's &[u8] over the mapping, were still alive,
writing the same file through the editor's descriptor. Nothing read the
mapping during the writes, but a shared slice whose memory changes
underneath it is undefined behaviour under Rust's aliasing rules.

Image::into_plan now detaches the edit's writes (patches, end of
allocation) into a Plan that owns all of its bytes and borrows nothing;
edit() takes the user-block size, drops the File — unmapping the file —
and only then commits the Plan. The invariant is documented in the image
module and the editor's module docs.

Test: edit::tests::file_is_not_mapped_while_an_edit_writes_it checks
/proc/self/maps at the moment each commit starts (write, resize, set_attr):
never mapped. With the commit moved back before the reader is dropped (the
previous order) it reports all three commits with the file mapped.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 14:14:58 -05:00
osobhandClaude Opus 5.5 b668878129 clawhdf5: FileEditor reports filters it cannot run as Error::Unsupported 
A dataset whose filter this build cannot encode (scale-offset, N-Bit, SZIP;
a plugin filter the build lacks) failed with Error::Format("unsupported
filter: 6"), although the editor documents every refused edit as
Error::Unsupported, and the Python bindings raised ValueError rather than
NotImplementedError. Every edit now maps FormatError::UnsupportedFilter to
Error::Unsupported; the file is left untouched as before.

Test: edit_interop unencodable_filters_are_unsupported — h5py scale-offset
datasets (integer with chunks, integer never written, float D-scale):
Error::Unsupported naming the filter, and the file byte for byte unchanged.
Fails on the previous editor (Format(UnsupportedFilter(6))).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 14:13:45 -05:00
osobhandClaude Opus 5.5 485bea0f4f clawhdf5: set_attr adds the Attribute Info message a version-2 header needs 
libhdf5 counts a version-2 object header's attributes through its Attribute
Info message (0x15) and reports none when the header has none. set_attr gave
v110/latest groups, the root group and datasets without attributes an
attribute message only, so h5py listed the attribute but len(obj.attrs) and
H5Oget_info's num_attrs said 0, and stayed wrong after h5py r+ added more.

Like H5O__attr_create, the edit now adds the message when a version-2 header
lacks it, in the same planned edit: version 0, the header's creation-order
track/index flags, maximum creation index 0, undefined fractal heap and
B-tree addresses, message flag DONTSHARE — byte for byte what libhdf5
writes. It goes before the attribute (libhdf5's order) when free space
holds both, else after it, so a continuation chunk made for the attribute
also takes it.

Test: edit_interop attribute_count_in_version_2_headers — v110 and latest
files, attributes set on the root group, groups and datasets with and
without existing attributes: h5py's len/num_attrs/list/values, h5dump -A
and our reader agree, also after h5py r+ adds attributes up to and past the
compact limit. Fails on the previous editor (h5py len 0).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 14:12:35 -05:00
osobhandClaude Opus 5.5 1ea9132e10 docs: changelog note for B-tree v2 internal node checksums 
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 14:11:47 -05:00
osobhandClaude Opus 5.5 04a7f6f6c7 read: of two links with one name, the first wins everywhere 
A valid group has one link per name, but a damaged or hand-made one can
have two. resolve_child followed the first soft link of the name, the
listing skipped a dangling one and listed the name via a later link, and
path resolution followed the last symbolic link: three answers. All now
take the first link of the name (header message order in a compact group,
name index order in a dense one) and ignore the rest, even if the first
dangles. That is libhdf5's rule for compact groups (H5G__compact_lookup
stops at the first Link message); h5py opens nothing for a dangling first
link although a later one resolves. For a dense group libhdf5
binary-searches the index and may land on another of several exact
duplicates; documented on first_link_named. find_symbolic_link's v2 branch
was dead (only v1 groups reach it) and is now v1-only.

Test: an h5py compact group with soft links dup_A (dangling, or to /d) and
dup_B (the other), dup_B renamed to dup_A in the header and re-checksummed.
Lookup, path and listing through all three readers match h5py for both
orders. With the old group_v2.rs the path lookup returned 42 where h5py
opens nothing.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 14:11:47 -05:00
osobhandClaude Opus 5.5 5b3d32b37d format: test the address overflow path on 64-bit hosts 
addr::to_usize's error branch only ran where usize is narrower than u64,
and no such target runs tests in CI, so on x86_64 the test checked only
that every u64 fits. to_usize and saturating_usize are now the usize
instances of generic to_index/saturating_index; the test runs the same
code with u32 standing in for a 32-bit usize: values past u32::MAX
(including one an `as` cast would wrap to 0x1234) are Overflow, and the
saturating form clamps. A mutant that truncates instead fails the test;
the old addr.rs does not provide the helper the test needs.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 14:09:37 -05:00
osobhandClaude Opus 5.5 f7e2ab12f2 clawhdf5: FileEditor skips optional filters that fail, as libhdf5 does 
The editor stored every chunk through the whole pipeline with filter mask 0.
For LZF that did not shrink a chunk, h5py instead stores it raw with the
filter's mask bit set. A chunk the editor stored LZF-encoded at exactly the
raw size was then rewritten raw by libhdf5 at the same size; libhdf5 does not
touch the index entry when the size is unchanged, so the stale mask 0 stayed
and h5py (and h5dump) could no longer read the dataset.

clawhdf5_format::filters::compress_chunk_masked runs the pipeline as
H5Z_pipeline does: an optional filter (H5Z_FLAG_OPTIONAL) that fails is
skipped and its bit set, a mandatory one fails the write, and LZF/Blosc
output no smaller than the input counts as failure, as in the reference
filters (their output buffer is the input's size). Deflate, LZ4, Zstd,
bitshuffle and bzip2 never fail on size in libhdf5 and are kept as before.

Test: edit_interop optional_filters_that_fail_are_skipped — the reviewer's
repro at every libver: the editor stores the chunk exactly as h5py does
(mask 1, size 5; shuffle+LZF+fletcher32 mask 2), h5py r+ rewrites and
extends the datasets, and h5py, h5dump and our reader read every value.
Fails on the previous editor (mask 0; h5dump cannot read /u8).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 14:09:34 -05:00
osobhandClaude Opus 5.5 b6cbd2319f format: checked chunk addresses on the parallel read path 
Three `chunk_info.address as usize` casts behind the `parallel` feature
survived the conversion, because check-32bit-casts.sh linted only default
features plus plugin-filters. On a 32-bit target with rayon a chunk address
past 4 GiB still wrapped onto another part of the file. They go through
addr::to_usize now, and the lane index (h % n, always < n) through
saturating_usize.

The script now lints no default features, default features, and every
optional feature but szip (wasm32; the set with zstd, which does not build
for wasm32, on the host, where the lint reports the same casts). With the
old parallel_read.rs/lane_partition.rs it fails listing the four casts; the
old script passed them. CHANGELOG and the design note give the exact count
(119) and what is not covered.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 14:08:57 -05:00
osobhandClaude Opus 5.5 92c8285549 format: verify B-tree v2 internal node checksums 
Only leaves and the header were checked. Harmless while every lookup read
the whole tree, but the indexed lookup prunes children by the keys stored
in internal nodes, so one corrupted byte there could route a name to the
wrong child and report it missing with no error. A BTIN whose lookup3
checksum does not match is now ChecksumMismatch on every read (lookups and
full traversals), as in libhdf5.

Test: one byte of the root BTIN of the 35 001-link h5py group's name index
changed -> lookups, paths and listings through File, MmapFile and LazyFile
all fail with ChecksumMismatch, and h5py refuses both. Before, lookups
returned Ok.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 14:07:01 -05:00
osobhandClaude Opus 5.5 d2b25f154f format: in-memory fast path in the Storage read helpers 
read_exact_at and read_upto (and so Window::read) ask as_contiguous()
first and slice the file directly when the backend holds it in memory:
one dynamic call per structure read instead of two or three (len,
read_at, then len again for errors). Same results and errors.

Provisional (busy machine, not for docs): a listing that walks 400
symbol-table groups through the facade went from about 18% to about 14%
slower than before the Storage conversion; the extra cost is a few tens
of nanoseconds per structure read, which the facade's per-lookup
re-listing (range-reads.md M0) multiplies.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 13:41:26 -05:00
osobhandClaude Opus 5.5 85efde0b4a test: rustfmt the lookup tests 
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 13:36:02 -05:00
osobhandClaude Opus 5.5 677dc5ec7c docs: FileEditor — changelog, limits and leaked space, README example 
known-issues records what the editor refuses, that freed space is never
reused (append-workload file sizes measured 2026-09-26 on tank with the
ignored measure_append_waste test; sizes are deterministic), and that there
is no journal.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 13:35:06 -05:00
osobhandClaude Opus 5.5 3c89a31df0 clawhdf5: FileEditor modifies existing files in place 
New clawhdf5::FileEditor opens an HDF5 file (h5py-written at any libver,
HDF5 2.0 format included, or clawhdf5-written) under an exclusive flock
and changes only what an edit touches:
- write_selection/write_all/write_values: compact, contiguous (also
  late-allocated) and chunked datasets, any selection. Chunks are decoded,
  updated and re-encoded; a filtered chunk that no longer fits moves to the
  end of the file unless it is the file's last structure, which grows in
  place. New chunks go into v1 B-tree, Extensible Array (paged data blocks
  included), Fixed Array and single-chunk indexes, created on first use.
- resize: grow chunked datasets up to maxshape.
- set_attr: add/replace compact attributes, in a NIL slot or a new
  continuation chunk.
Each edit is planned in an in-memory image and refused whole
(Error::Unsupported) when any part is unsupported (v2 B-tree / implicit
new chunks, shrinking, vlen/reference data, dense or order-tracked
attributes, cache images, paged/persistent free space). Commit writes and
syncs new space before patching existing bytes. Layout v5 (HDF5 2.0)
array indexes use 8-byte filtered chunk sizes, as libhdf5 does.

Error gains Unsupported/InvalidArgument/Locked and is #[non_exhaustive];
the Python bindings map them. build_attr_message is public.

Tests (h5py, h5dump, h5rs check --data after every round; h5py r+
afterwards): appends crossing EA super/data blocks and B-tree splits, the
same B-tree node counts and EA statistics as libhdf5 for the same writes
(in order, reversed and shuffled; paged blocks), every layout and chunk
index overwritten under random selections, attributes to continuation
chunks, random operations against a model, refused edits leave the file
byte-identical, locking.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 13:35:06 -05:00
osobhandClaude Opus 5.5 1b4a93f65a docs: indexed name lookups and checked address conversion (M0) 
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 13:33:24 -05:00
osobhandClaude Opus 5.5 b41583113a format: no truncating u64 -> usize casts 
Every `u64 as usize` cast in clawhdf5-format (115 on wasm32) now goes
through addr::to_usize for values read from the file — addresses, lengths,
counts, dimensions: FormatError::Overflow where the value does not fit
instead of wrapping onto another part of the file on a 32-bit target — or
addr::saturating_usize for counts bounded by something in memory (codec
progress counters, writer sizes), which fail a bounds check or allocation
rather than wrap. A chunk whose offset does not fit lies outside the
dataset and is skipped; partial reads treat such an offset as out of the
buffers. On 64-bit targets nothing changes.

scripts/check-32bit-casts.sh (run by ci-test.sh) lints the wasm32 build
with clippy's cast_possible_truncation and fails on any u64 -> usize
finding; before this commit it listed 115.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 13:33:24 -05:00
osobhandClaude Opus 5.5 02e89c1d2d read: look names up through the dense name indexes 
Finding one link or attribute by name read every entry: Group::dataset and
Group::group (File, MmapFile, LazyFile) listed the whole group per call, and
path resolution scanned each group's links. Opening every child of a
35 001-link group by name decoded ~1.2e9 links.

Now a dense group's v2 B-tree name index (type 5, lookup3 hash of the
name) is descended to the records with the name's hash
(btree_v2::find_btree_v2_records reads only the nodes whose key interval
overlaps), and only those links are read and compared; all hash-equal
records are compared, so libhdf5's tie order does not matter. Dense
attributes the same through their type 8 index
(attribute::find_attribute_in_file, facade attr(name)); huge heap objects
through their ID-ordered index. group_v2::resolve_child returns what the
listing has under a name (soft links followed, dangling/external ones not
found). Group::entries and File::group_at hand out a listing's addresses.

The lookup-stats feature counts heap objects read. Tests: one lookup in
an h5py-written 35 001-link group with colliding hashes reads at most two
links (before: 35 001, failing), attribute lookups likewise (before: 3 000,
failing), every child opens through all three readers and matches h5py,
every link kind resolves as h5py resolves it in dense and compact groups,
300 huge attributes are found, and a range search matches a full scan at
every tree depth.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 13:33:19 -05:00
osobhandClaude Opus 5.5 5d17712adb docs: changelog and design status for range-read milestone M1 (part 1) 
Lists the parsers now reading through Storage, what still needs the
whole file (v2 B-tree-indexed structures: a clean error; raw data: M2),
the equivalence harness, and the evidence that nothing changed: existing
tests, a byte-identical conformance results.json and per-file probe
output against f2ff2c4, and identical slice-API transcripts over 748
files between the two builds.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 13:29:11 -05:00
osobhandClaude Opus 5.5 476960f4b8 format: fix a clippy lint in the global heap test fixture 
With Storage in scope, `data.len()` on a `&&[u8]` resolves to
Storage::len (already a u64), so `as u64` was a no-op cast; name the
slice method explicitly.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 13:11:46 -05:00
osobhandClaude Opus 5.5 24f0c71939 format: keep dataset_fill_value_in's &[u8] signature 
Making dataset_fill_value_in generic over Storage broke callers that pass
an array (`include_bytes!`): a generic parameter does not unsize-coerce
`&[u8; N]`. It takes &[u8] again, as before this branch, and wraps the
new dataset_fill_value_from_storage(&dyn Storage, ..).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 13:11:46 -05:00
osobhandClaude Opus 5.5 5705866d40 format: equivalence harness for the Storage migration 
tests/storage_equivalence.rs walks real files and runs every metadata
parser converted to Storage twice per object — over the file as a slice
and over a read_at-only CountingStorage — and requires identical results,
values and errors alike; the only allowed difference is the clean
ContiguousStorageRequired from the structures still indexed by a v2
B-tree (dense attributes, a SOHM B-tree index), which is counted. It
covers superblock and extension, cache image, SOHM table/list/B-tree,
object headers, attributes, fill values, shared messages, symbol-table
groups (local heap, B-tree, nodes, names), fractal heaps and their
objects, VDS mappings, and fixed/extensible array chunk indexes.

Inputs: every fixture; files h5py writes for what the fixtures lack
(extensible arrays with super blocks and paged data blocks, paged fixed
arrays, a 400-group v1 file with a user block, a 300-link dense group,
dense, shared and committed-type attributes, SOHM list and B-tree
indexes; honours CLAWHDF5_PYTHON / CLAWHDF5_REQUIRE_INTEROP); and, with
CLAWHDF5_STORAGE_CORPUS set, a corpus such as conformance/.cache/corpus.
Milestones M2/M3 add their parsers to check_object.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 13:09:32 -05:00
osobhandClaude Opus 5.5 2b7065998a docs: ZFP reads (changelog, README feature table, known issues) 
ZFP (32013) was the one plugin filter still listed as UnsupportedFilter.
Conformance on tank, `conformance/run.sh --no-fetch` (2026-09-26): 600 of
697 files ok (baseline 599); h5ex_d_zfp.h5 is newly ok.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 13:08:47 -05:00
osobhandClaude Opus 5.5 6a535e2651 format: resolve virtual dataset mappings over Storage 
DataLayout::resolve_vds_mappings_in reads the global heap collection
holding a virtual dataset's mappings through GlobalHeapCollection::
parse_in; resolve_vds_mappings is a wrapper. The rest of the data layout
module parses message payloads and does not read the file.

New test: the virtual dataset of vds_same_file.h5 resolves to the same
mappings through a read_at-only CountingStorage, in two reads.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 13:06:12 -05:00
osobhandClaude Opus 5.5 a65a2b7f18 format: resolve fill values over Storage 
dataset_fill_value_in now takes the file as any Storage (generic, so
every &[u8] caller compiles unchanged) and follows a shared fill value
message through message_data_with_sohm_in. The raw-data helpers in the
module (read_full_with_fill, apply_to_unallocated_chunks) walk chunk
indexes and stay on &[u8] until milestone M2.

New test: the four datasets of shared_fill_value.h5, two with their
fill value in the SOHM heap, resolve identically through a read_at-only
CountingStorage.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 13:06:12 -05:00
osobhandClaude Opus 5.5 2c292404d2 format: read attributes over Storage 
AttributeMessage::parse_in_storage, extract_attributes_full_in and
extract_attributes_tolerant_in take the file as &dyn Storage: shared
datatypes, dataspaces and attributes are resolved through the Storage
shared-message path, and dense attributes' fractal heap through
FractalHeapHeader::parse_in / read_managed_object_in. The dense-storage
name index is a v2 B-tree, which is not read over Storage yet: over a
backend without the whole file in memory it is a clean
ContiguousStorageRequired error, never a partial list. The &[u8]
functions are wrappers.

New test: every object in five h5py-written fixtures (compact, shared
and dense attributes) reads identically through a slice as Storage, and
through a read_at-only CountingStorage except the dense ones, which give
the clean error.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 13:05:48 -05:00
osobhandClaude Opus 5.5 ba3f476be6 format: resolve shared messages over Storage 
Every shared-message entry point has an *_in(&dyn Storage, ..) core:
message_data(_with_sohm), resolve_shared_message(_with_sohm),
resolve_sohm_message, load_sohm_table, and the SMTB/SMLI parsers, which
read the signature and then all entries in one bounded read (the list's
open-ended &file_data[pos..] slice is gone). Object headers and the SOHM
fractal heap are read through their Storage parsers; a SOHM B-tree index
still needs the v2 B-tree over a slice, so over a backend without one it
is a clean ContiguousStorageRequired error. New signature::find_signature_in
probes the candidate offsets with 8-byte reads. The &[u8] functions are
wrappers.

New tests: SOHM tables and lists with 4- and 8-byte offsets, at two
offsets, cut at every length and with a bad signature, parse identically
through a read_at-only CountingStorage in at most two reads; the
signature search matches the slice search.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 13:02:58 -05:00
osobhandClaude Opus 5.5 ff6d644391 format: read extensible array chunk indexes over Storage 
ExtensibleArrayHeader::parse_in reads the header in one read, and
read_extensible_array_chunks_in reads each index block, super block and
data block as a prefix read and then one window of the whole structure
(paged data blocks included); checksums and elements are checked in the
window with bounds errors reported as the whole-file checks did. The
&[u8] functions are wrappers. New test: an array with inline elements
and a data block, cut at every length and damaged in each structure,
reads identically through a read_at-only CountingStorage.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 13:01:11 -05:00
osobhandClaude Opus 5.5 cf2b408a63 format: read fixed array chunk indexes over Storage 
FixedArrayHeader::parse_in reads the header in one read, and
read_fixed_array_chunks_in reads the data block's prefix and then the
whole block, paged or not, as one window; checksums and elements are
checked in it, with bounds errors reported as the whole-file checks did
(also in builds without the checksum feature, where the per-element
checks are the only ones). The open-ended &file_data[offset..] slices are
gone. The &[u8] functions are wrappers. New test: non-paged and paged,
filtered and unfiltered arrays, cut through the data block and with
damaged bytes, read identically through a read_at-only CountingStorage.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 12:57:08 -05:00
osobhandClaude Opus 5.5 d97b3d703a facade: zfp feature; ZFP read bit-exact against libhdf5 + libzfp 
The facade forwards `zfp` and adds it to `plugin-filters`.
tests/zfp_interop.rs: h5py + hdf5plugin (H5Z-ZFP 1.1.1, zfp 1.0.1)
write 2205 datasets over 16 modes (rate, precision, accuracy,
reversible, expert settings at their edges) x int32/int64/float/double
x 1-4-D shapes with partial edge chunks, partial blocks and unit chunk
dimensions x smooth/noisy/wide-range/zero/inf-NaN data; clawhdf5 must
read each byte for byte as h5py does (read back after closing the file:
h5py returns a chunk still in libhdf5's cache without decoding it). A
second test swaps a file's header words to what a big-endian writer
stores and checks the byte-swapped values match h5py's.

The left-out-filter test now expects ZFP only in builds without it; CI
lints `zfp` alone and runs the new test with the plugin-filter step.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 12:57:02 -05:00
osobhandClaude Opus 5.5 23a4784e72 format: ZFP decoder (filter 32013), pure Rust, read-only 
A port of the zfp 1.0.1 decoder and of H5Z-ZFP 1.1.1's decompression
path behind a new `zfp` feature (in `plugin-filters`): every mode
(fixed rate, precision, accuracy, reversible, expert), int32, int64,
float and double, 1-4 dimensional fields with partial blocks, and
headers written big-endian (values byte-swapped as H5Z-ZFP does).

H5Z-ZFP keeps the zfp header in cd_values (version word, then the
magic/metadata/mode bit stream); each chunk is the bare stream. The
decoder reproduces libzfp bit for bit: integer arithmetic wraps as
libzfp's, block exponents scale by exact powers of two, and the mode
goes through zfp_stream_mode as H5Z-ZFP hands it to zfp. A stream that
ends early is an error (libzfp reads past its buffer), as is a field
whose size is not the chunk's; the output is allocated only once the
stream holds a bit per block.

Tests: header/mode unit tests, and a counting-allocator fuzz of random
headers (expert parameters at their edges) and streams.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 12:56:07 -05:00
osobhandClaude Opus 5.5 a0160730f2 format: read fractal heaps over Storage 
FractalHeapHeader::parse_in reads the header as one window (a second,
longer one when it holds an I/O filter pipeline); read_managed_object_in
reads direct blocks, indirect blocks (one window up to the last child
entry) and huge objects with bounded reads. The &[u8] methods are
wrappers. A huge object indexed by the huge-object v2 B-tree, which is
not converted yet, is a clean ContiguousStorageRequired error on a
backend without the whole file in memory (after the "no index" check,
so the error order is unchanged).

storage::Window (crate-internal) reads a window of a structure and
reports bounds failures exactly as the whole-file ensure_len did, and a
short read inside the file is now a Storage error rather than an EOF.

New tests: headers (with and without a filter pipeline) cut at every
length, and managed objects in a direct root and through an indirect
root, huge objects with direct IDs and tiny objects, give identical
results through a read_at-only CountingStorage.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 12:54:48 -05:00
osobhandClaude Opus 5.5 24cbf12f16 format: read group B-tree (v1) nodes over Storage 
BTreeV1Node::parse_in reads a node's header and then its keys and
children, two bounded reads; collect_symbol_table_nodes_in walks the tree
over any Storage. The &[u8] functions are wrappers. New test: nodes with
siblings and 4- and 8-byte offsets cut at every length, and a two-level
tree with truncated leaves, give identical results through a read_at-only
CountingStorage (six reads for the three nodes).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 12:51:47 -05:00
osobhandClaude Opus 5.5 bcf3ae4856 format: read symbol table nodes over Storage 
SymbolTableNode::parse_in reads the node's 8-byte header, checks the
entries against the end of the file as before, and reads them in one
bounded read. parse is a wrapper. New test: nodes with 4- and 8-byte
offsets, valid and with a bad version, at two offsets and cut at every
length, parse identically through a read_at-only CountingStorage.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 12:51:47 -05:00
osobhandClaude Opus 5.5 06625b7470 format: read global heap collections over Storage 
GlobalHeapCollection::parse_in / parse_index_in read the header, check
the collection against the end of the file, and read the collection in
one bounded read; objects are indexed in it with file offsets, as before.
The &[u8] functions are wrappers. New test: collections with 4- and
8-byte lengths, one whose size runs past the file and one whose object
overruns it, at two offsets and cut at every length, give identical
results through a read_at-only CountingStorage.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 12:50:25 -05:00
osobhandClaude Opus 5.5 aab7ea9e8f format: rustfmt the Storage conversions 
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 12:49:05 -05:00
osobhandClaude Opus 5.5 6a9bb02f37 format: read local heaps over Storage 
LocalHeap::parse_in reads the header in one bounded read,
validate_free_list_in reads each free block's two lengths, and
read_string_in reads from the string to the end of the data segment
once and looks for the terminator there. The &[u8] methods are wrappers.
New test: a heap without free space, with a valid free block and with a
free block overrunning the segment, cut at every length, parse, validate
and read strings identically through a read_at-only CountingStorage.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 12:49:05 -05:00
osobhandClaude Opus 5.5 0d908facd3 format: read the superblock extension and cache image over Storage 
read_superblock_extension_in, cache_image_state_in, CacheImage::decode_in
and CacheImage::block_in take &dyn Storage (whose length is the end of
file); the image block is one bounded read. The &[u8] functions are
wrappers; applying an image in place still needs the bytes in memory.
New test: extension messages, a cache image and a corrupt one decode to
the same results through a read_at-only CountingStorage.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 12:47:59 -05:00
osobhandClaude Opus 5.5 cd828725c7 format: parse object headers over Storage 
ObjectHeader::parse_in(&dyn Storage, u64, ..) reads the signature, the
prefix (a window of at most 34 bytes for version 2) and then each chunk,
continuation chunks included, as one bounded read; the message loops run
unchanged on the chunk with chunk-relative positions. parse keeps its
&[u8] signature as a wrapper. Bounds errors are reported as before, with
absolute positions and the file's length. New test: headers of both
versions, with times, phase-change values, creation order and a
continuation chunk, and every truncation of each, parse identically
through a read_at-only CountingStorage.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 12:46:51 -05:00
osobhandClaude Opus 5.5 512a6a753f format: parse the superblock over Storage 
Superblock::parse_in(&dyn Storage, offset) reads one bounded window of
128 bytes (the largest superblock is 100) and runs the existing version
parsers on it; parse and refresh_eof keep their &[u8] signatures as
wrappers. No behaviour change: on a file longer than the window no bounds
check can fail, and on a shorter one the window is the whole file. New
test: every version and truncation parses to the same result through a
read_at-only CountingStorage, in one read.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 12:44:52 -05:00
osobhandClaude Opus 5.5 6a4707d791 format: add the Storage trait; make the error enums non-exhaustive 
Range-read milestone M1, first step (docs/design/range-reads.md §3(a)):
a synchronous, no_std read interface with u64 offsets, read_at returning
Cow<[u8]>, read_ranges, len and an as_contiguous fast path. Implemented
for [u8], Vec<u8>, &T, Box<T> and Arc<T>; slices serve borrowed bytes.
read_exact_at reproduces the parsers' UnexpectedEof bounds error exactly,
so converted modules keep their error values.

FormatError gains Storage(String) and ContiguousStorageRequired; it and
the facade Error are now #[non_exhaustive] (breaking for exhaustive
matches, noted in the changelog; the Python bindings' match gets a
wildcard arm).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 12:44:36 -05:00
osobhandClaude Opus 5.5 6248b411f0 format: one checked conversion from file address to index 
addr::to_usize turns a 64-bit file address or length into a slice index,
failing with FormatError::Overflow where it does not fit usize (32-bit
targets such as wasm32) instead of truncating like an `as usize` cast.
Callers are converted in the following commits.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 12:43:49 -05:00
osobh 479d8b47e0 Merge pull request 'Chunked reads beat an h5py process pool; unlimited writer B-trees; Blosc2; 599/697 conformance' (#16) from feat/p2b-scale into main
CI / test-arm64 (push) Successful in 1m27s
Details
CI / test (push) Successful in 17m32s
Details 
Reviewed-on: #16
 2026-09-26 17:42:15 +00:00
osobhandClaude Opus 5.5 f2ff2c424f bench: chunked full reads now beat an h5py process pool
CI / test-arm64 (pull_request) Successful in 1m25s
Details
CI / test (pull_request) Successful in 13m0s
Details 
Idle-start run on tank at c5334b1 (noisier than the last: compare ratios
within the run). Full reads of deflate data at 16 threads: 4944 MB/s vs
3135 for 16 h5py processes (1.58x; 0.69x-0.76x before). One thread with
the default pool: 6143 MB/s, 15x one h5py call. The concurrent-read
known issue is closed.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 12:20:47 -05:00
osobhandClaude Opus 5.5 c5334b1c97 docs: conformance report after this batch (599 of 697 ok) 
Regenerated on tank: ok 575 -> 599, our-error 10 -> 4, mismatch 20 -> 2,
no panics, hangs, crashes or OOM. Newly ok: metadata-cache-image files,
the CVE open-time checks, v1 chunk-key lookup as libhdf5 does it, the
scale-offset fixes, Blosc2, and harness corrections. Baseline raised.
h5rs check --data flags none of the 435 fully-read ok files.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 12:10:57 -05:00
osobh d0e3beb3aa Merge branch 'docs/p2b-range-read-design' into feat/p2b-scale 2026-09-26 11:57:35 -05:00
osobh 9a73299594 Merge branch 'fix/p2b-remaining-conformance' into feat/p2b-scale 
# Conflicts:
#	CHANGELOG.md
#	crates/clawhdf5-format/src/chunked_read.rs
 2026-09-26 11:57:35 -05:00
osobh 4b02e7d068 Merge branch 'feat/p2b-blosc2' into feat/p2b-scale 
# Conflicts:
#	CHANGELOG.md
 2026-09-26 11:57:11 -05:00
osobh d7f07fa5c1 Merge branch 'feat/p2b-writer-btree-internal-nodes' into feat/p2b-scale 
# Conflicts:
#	CHANGELOG.md
 2026-09-26 11:57:11 -05:00
osobh bdb2c0e36b Merge branch 'perf/p2b-chunked-full-reads' into feat/p2b-scale 2026-09-26 11:57:11 -05:00
osobhandClaude Opus 5.5 55e0e7e9cf docs: conformance numbers after the review fixes (598 of 697 ok) 
cve-2025-44905 now reads as h5py reads it (the v1 chunk B-tree lookup),
leaving 5 our-errors: cve-2025-2308, cve-2025-44904 and
bad_nbit_parms_walk (corrupt data HDF5 2.0 reads through a bug), and
the Blosc2 and ZFP filters. The five unloadable-cache-image files stay
ok, now with the library behaving as the probe reports.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 11:53:46 -05:00
osobhandClaude Opus 5.5 00f94d57ed test(io): read the cache-image fixture's values with as_chunks 
clippy (with the async feature) flags chunks_exact with a constant size
in the AsyncHDF5File cache-image test added in d493d47.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 11:52:06 -05:00
osobhandClaude Opus 5.5 4c01267b76 test: every scale-offset dataset h5py writes reads as h5py reads it 
The scale-offset fix (d110b1d) was covered only by unit vectors from
CVE chunks, and documented as three corner cases. The review found it
is much bigger: of 1480 scale-offset datasets h5py writes (every
integer type i1..u8, f4 and f8, both byte orders, with and without a
fill value, scaleoffset 0..full width), v2.7.0's decoder read 332
differently from h5py: 151 returned wrong values with no error (82
integer datasets with scaleoffset=0 and a wide range, 51 full-width
i4/u4/i8/u8, 18 f4 D-scale datasets with a large range) and 181 failed
to read. The cause in every case is a chunk libhdf5 stores at full
width, whose elements were decoded as offsets from minval.

tests/scaleoffset_interop.rs generates that matrix with h5py at test
time, stores h5py's decoded values uncompressed next to it, and
compares every dataset's bytes. It passes on this branch; with the
filters.rs before d110b1d it reports "332 of 1480 scale-offset datasets
differ from h5py".

CHANGELOG: a Correctness entry stating this was silent wrong data in
every release that decoded scale-offset (v2.2.0 to v2.7.0), replacing
the corner-case wording. docs/known-issues.md: a fixed entry with the
affected cases.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 11:51:30 -05:00
osobhandClaude Opus 5.5 d493d4792e fix: apply the superblock extension and cache image in every opener 
File, MmapFile and LazyFile decoded the superblock extension and laid a
metadata cache image over the file's metadata; the other readers did
not, so the same file read differently by entry point: NativeVol,
AsyncHDF5File and MpiVol (clawhdf5-io) and the external source files of
a virtual dataset (clawhdf5-format vds.rs) read a file with an image
from its own bytes, which libhdf5 does not (they may be stale, or zeros:
h5clear_mdc_image.h5 failed with InvalidObjectHeaderVersion(0)), and
skipped the extension checks File::open makes (cve-2020-10810/10812).

Each of them owns its buffer, so each now calls the shared
superblock_ext::apply_cache_image_in_place, which checks the extension
and writes the image's entries in place (only the image block is
copied). These readers read whole datasets and cannot open a file and
fail each object, so an image libhdf5 cannot load is refused with the
image's error, never read around. clawhdf5-io's vol::load_hdf5 wraps it
for NativeVol (at open; for from_bytes the error is reported on read,
as a truncated file already was) and MpiVol. The MpiVol edit is minimal
and was not compiled: the mpi-io feature needs an MPI installation this
machine does not have (mpi-sys's build script panics).

Tests: NativeVol (open_path and from_bytes), AsyncHDF5File and a VDS
whose source file is h5clear_mdc_image.h5 (vds_interop.rs, against
h5py) read the fixture's values; the corrupted-image variants are
refused. Each fails without its fix.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 11:49:13 -05:00
osobhandClaude Opus 5.5 6559a91495 fix: open a file whose cache image cannot load, and fail its objects 
For a metadata cache image libhdf5 cannot load, libhdf5 opens the file
and fails the first metadata read (the image loads on the first
H5C_protect after open); h5py reports the error on the root group. The
conformance probe reported it that way, but File::open refused the file,
so the gate counted cve-2025-6269-1..4 and cve-2025-6516 as agreeing
with h5py for behaviour the library did not have.

The library now behaves as the probe reports: File (mmap, buffered and
from_bytes) and MmapFile open the file and every object lookup (dataset,
dataset_at, group, group listings and attributes, VL decoding) fails with
the image's error; LazyFile reads the root group's header at open, so
its open is that first read and fails. Probe and library take the
three-way decision (refuse at open / image loads / image cannot load)
from the same clawhdf5_format::superblock_ext::cache_image_state.

One deliberate difference from libhdf5 remains, documented: after the
failed first read libhdf5 reads the file's own metadata, which the image
was meant to replace and may be stale; here every lookup keeps failing.
File::cache_image_error / MmapFile::cache_image_error expose the error
to code that parses as_bytes() itself; h5rs checks it before reading any
object header (h5rs ls on cve-2025-6269-1 said "invalid object header
version: 0" from the stale bytes).

Test: metadata_cache_image.rs an_image_libhdf5_cannot_load_fails_every_object
(the fixture with its image signature broken; h5py opens that file and
fails the first read with "Bad metadata cache image header signature").
It fails on the previous commit, where File::open refuses the file.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 11:46:28 -05:00
osobhandClaude Opus 5.5 60502593b7 fix: apply a metadata cache image without copying the file 
apply_cache_image returned a copy of the whole file with the image's
entries written in, and File (mmap by default), MmapFile and LazyFile
used that copy for every read: opening a 1 GiB sparse file with an image
needed 2 GB of memory, and an 8 GiB one aborted the process, where
de2a53f (which ignored the image) opened them in a few MB.

The metadata parsers read one contiguous slice, so the image still has
to be laid over the file's bytes; it is now laid over a private copy
that costs only the pages it touches:

- clawhdf5_format::superblock_ext::CacheImage decodes the image into an
  entry list (address, offset in the block, length) and applies it to
  any destination; cache_image_state tells an opener whether the file
  has no image, a loadable one, or one libhdf5 cannot load;
  apply_cache_image_in_place is for readers that own their buffer.
  apply_cache_image and metadata_view (which copied) are gone.
- clawhdf5_io::HDF5Read::private_copy returns a writable private copy
  of a reader's bytes: MmapReader gives a MAP_PRIVATE copy-on-write
  mapping (memmap2 map_copy), so only the pages the entries land on are
  copied; the default copies the bytes (in-memory readers).
- File, MmapFile and LazyFile write the image into that mapping
  (crate::cache_image). File::from_bytes / open_buffered patch their own
  buffer in place, copying only the image block, as libhdf5 does. A
  file without an image is read straight from the mapping, unchanged.

An image entry that runs past the end of file is now refused: libhdf5
checks only that it starts inside the file, and the images libhdf5
writes never do this, but those bytes have nowhere to go in a view of
the file.

Tests: tests/cache_image_memory.rs has libhdf5 (through ctypes) add an
image to a 1 GiB sparse file and bounds resident-memory growth for all
three openers at 256 MiB; it fails on the previous commit (File::open
grew 2,148,720,640 bytes). reader.rs zero_copy_tests check that a file
without an image is read from the mapping itself and that an image goes
into a copy-on-write mapping, not a heap copy; clawhdf5-io checks that
private_copy writes never reach the reader or the file.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 11:42:43 -05:00
osobhandClaude Opus 5.5 a6ed3a5c7d fix(format): resolve cache-image flush-dependency parents as libhdf5 does 
The review suggested libhdf5 loads every image entry and resolves
flush-dependency parents afterwards. It does not:
H5C__reconstruct_cache_contents (HDF5 1.14.6 and 2.0.0, and develop)
inserts each entry and then searches the cache index for its parents in
the same loop, failing with "fd parent not in cache?!?" when one is
missing. So a parent must be an earlier image entry, as before, or
metadata cached before the image loads: the superblock (address 0) and
the superblock extension's object header, which libhdf5 reads to find
the image. Those two were refused as parents; they are now accepted.
A parent listed after its child is still refused, as libhdf5 refuses
it, and so is an entry that is its own parent ("Child entry flush
dependency parent can't be itself").

apply_cache_image takes the superblock to know the extension address.

Test: superblock_ext::tests::flush_dependency_parents_must_already_be_cached
(parent-first loads, child-first refused, extension header accepted,
self-parent refused); the extension-header case fails without the fix.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 11:36:10 -05:00
osobhandClaude Opus 5.5 3da118d2ee style(format): iterate the chunk index in BlockGeometry::place 
clippy's needless_range_loop, missed before the B2ND streaming commit.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 11:36:00 -05:00
osobhandClaude Opus 5.5 7515e5dcbd docs: CHANGELOG for the Blosc2 allocation bounds 
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 11:35:19 -05:00
osobhandClaude Opus 5.5 742ed4dfb8 fix(format): read a v1 chunk B-tree where libhdf5's lookup finds chunks 
libhdf5 does not walk the chunk B-tree to read a dataset: it looks each
chunk up (H5B_find with H5D__btree_cmp3 and H5D__btree_found), asking for
the element-size coordinate as 0. collect_chunk_info_checked now parses
the tree with its keys and returns each stored chunk only when that
lookup, replayed over the scaled keys, finds it.

A key with a non-zero element-size coordinate is therefore found in a
1-D dataset (cmp3 compares only the first coordinate there, and found
compares with <=) and missed in a dataset of rank 2 or more, which reads
fill values. The previous commit refused every such key, which refused
1-D files libhdf5 reads correctly; before that, the rank-2 case read the
chunk's data where h5py reads fill values (cve-2025-44905
/Shuffle_float_data_le, now identical to h5py, so it leaves the
conformance report's list of libhdf5 bugs).

Test: chunk_keys_with_an_element_offset_read_as_libhdf5_reads_them
compares 1-D and 2-D files against h5py's values. It fails on the
previous commit (the 1-D file is refused) and with the refusal removed
(the 2-D file reads 0..23 where h5py reads fill values).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 11:35:13 -05:00
osobhandClaude Opus 5.5 2ba4bc97d8 test(format): fuzz Blosc2 decoding for peak allocation 
The Blosc2 fuzz tests checked only for panics and the size of the output,
so the offsets-chunk amplification passed all 40,000 iterations.
tests/blosc2_alloc_bounds.rs now measures peak allocation (a counting
global allocator) and asserts it stays within 6x the HDF5 chunk size plus
twice the input plus 2 MiB (ruzstd's fixed state) for:

- 20,000 mutated fixture frames, decoded with their real chunk size as
  the limit, with edits aimed at the frame's and chunks' size fields;
- 20,000 mutated first chunks on their own;
- 5,000 frames built from random header sizes, offsets chunks and B2ND
  shapes (padding chunk and block shapes, chunks larger than the array,
  NaN, zero and repeated-value chunks).

Against the code before this series every test in the file fails (the
fuzz tests at frame iteration 3913, a zstd window, and random frame 289,
the offsets chunk); now the worst frame peaks at 0.48 of the bound.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 11:35:08 -05:00
osobhandClaude Opus 5.5 d9e4dfb6e6 fix(format): cap the Zstandard window at what the output can need 
ruzstd reserves a frame's declared window (up to its 100 MiB default)
when a decoder is reset for a new frame, before decoding anything. The
Blosc, Blosc2 and bitshuffle decoders reuse one decoder per chunk, so a
Blosc2 chunk of two 16-byte streams, each declaring a 96 MiB window,
allocated 128 MiB. zstd_decode_into now sets the decoder's maximum window
to twice the stream's output (at least 128 KiB): c-blosc, c-blosc2 and
bitshuffle compress each block in one call with its size known, so
libzstd's window never exceeds the block.

Found by tracking peak allocation in the Blosc2 fuzz test.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 11:34:28 -05:00
osobhandClaude Opus 5.5 22dc87b07c fix(format): never hold a B2ND chunk's padding 
B2ND chunks were decoded whole, padding included, with up to 16x the HDF5
chunk size as their limit, so a crafted frame made each chunk allocate and
fill up to 16x the output (4 GiB for a 256 MiB HDF5 chunk). The padding is
the real bound (prod(ceil(c/b)*b) per chunk), but that can be 2^ndim times
the array, so it is no longer held at all:

- A Blosc2 chunk is now decoded block by block (decode_blocks), each block
  handed to a sink as it is ready, with at most three blocks of scratch.
  blosc2_decompress_chunk and plain frames still collect every block.
- reassemble places each B2ND block straight into the output and skips
  blocks that are all padding (they are not decoded unless the delta
  filter needs the first block). A frame's NaN chunks are handed over one
  B2ND block at a time and its zero chunks cost nothing.
- A B2ND chunk must decode to exactly its padded size, its Blosc2 blocks
  must be whole B2ND blocks no larger than the output, and a chunk may not
  be larger than the array (hdf5-blosc2's chunk is the array), so a block
  is never larger than the output.

Peak allocation for a 10-D array padded to 13x (NaN, repeated-value and
stored-block chunks) and for a 16x chunk was 4.5 MB and 17.8 MB for
315 KB and 1 MiB outputs before, and is now within the tests' bound.
Blosc2 files written by hdf5plugin in 9-D and 12-D, an 8 MiB single chunk,
1x1x1 and edge-chunk shapes still read exactly.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 11:30:39 -05:00
osobhandClaude Opus 5.5 e05530a805 fix(format): an empty Blosc2 chunk no longer allocates its block size 
A chunk header with nbytes 0 and no special type kept its declared block
size (up to 512 MiB): the block size was clamped to nbytes only when nbytes
was positive, and the scratch blocks were allocated before the (empty)
block loop, so a 20-byte chunk allocated about 1 GiB. The block size is now
clamped to nbytes always, and an empty chunk returns before any scratch is
allocated.

A frame chunk must also decode to the size the frame header gives it
(chunksize, or the remainder for the last chunk), and is decoded with that
as its limit, so an empty chunk in a frame for a non-empty HDF5 chunk is an
error rather than an empty result.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 11:27:06 -05:00
osobhandClaude Opus 5.5 989335b67b fix(format): bound a Blosc2 frame's offsets chunk by the HDF5 chunk size 
parse_frame sized the offsets chunk from the frame header's own nbytes and
chunksize, so a 173-byte frame declaring 32 Mi chunks, with a 40-byte
repeated-value offsets chunk, built 256 MiB (up to 2 GiB) of offsets for a
1 MiB HDF5 chunk and then returned 4 bytes. The offsets chunk is now capped
at the output limit (at least 128 bytes); a frame whose nbytes/chunksize
imply more chunks than that is refused before anything is allocated.

tests/blosc2_alloc_bounds.rs measures peak allocation with a counting
global allocator; the reviewer's frame failed it (decoded Ok(4)) before.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 11:26:28 -05:00
osobhandClaude Opus 5.5 bf4aefcd00 format: a one-thread pool decodes on the caller only; keep 1 MiB scratch 
Review follow-ups. With run_with_helpers a one-thread rayon pool gave each
read a second core (the caller plus the worker), so --decode-threads 1 no
longer matched h5py's one core per call; such a pool now adds no helper.
Per-thread decode scratch is kept up to 1 MiB per buffer (was 4 MiB),
bounding what never-exiting pool workers hold.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 11:22:07 -05:00
osobhandClaude Opus 5.5 378afa1584 docs: changelog and known issues for the remaining conformance errors 
Conformance on tank, conformance/run.sh --no-fetch (2026-09-26): 597 of
697 ok, 6 our-errors (4 corrupt objects HDF5 2.0 reads through a bug, the
Blosc2 and ZFP filters), 2 mismatches (the known h5py big-endian VL bug).
Closes the known-issues entries for metadata cache images,
cve-2024-32624, cve-2020-10810/10812, and unfiltered chunks of the wrong
size; the N-Bit / 64-bit scale-offset entry is recorded as not our bug.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 10:40:18 -05:00
osobhandClaude Opus 5.5 67958b08d9 conformance: list the corrupt objects HDF5 2.0 reads through a bug 
Four of the remaining our-errors are objects the reference (h5py 3.16 /
HDF5 2.0) reads only through a libhdf5 bug, and clawhdf5 refuses:
cve-2025-2308 (scale-offset codes past the end of the chunk),
cve-2025-44904 (short unfiltered chunks), bad_nbit_parms_walk.h5 (an N-Bit
parameter list one value short; libhdf5's own test_filter_bad_params now
requires the read to fail) and cve-2025-44905 /Shuffle_float_data_le (a
chunk key libhdf5's lookup misses, reading fill values). report.py lists
them under Known not-our-bug and counts them in the summary; they stay
our-errors in the class counts.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 10:39:25 -05:00
osobhandClaude Opus 5.5 f512bf3d09 conformance: report a cache image libhdf5 cannot load where it does 
libhdf5 loads a metadata cache image when it first reads metadata (the
root group), not at open, so for cve-2025-6269-1..4 and cve-2025-6516 (all
corrupt images) h5py opens the file and fails on "/". The probe reported
the image's error as an open error, which made those files our-errors;
it now records it on the root object, where h5py reports it. File::open
still refuses such a file outright: nothing in it can be read.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 10:38:50 -05:00
osobhandClaude Opus 5.5 8295d01614 format: a chunk offset past usize writes nothing 
Placing a chunk cast its u64 offsets to usize; on a 32-bit target an
offset past the address space wrapped into the output (and could then
overlap another chunk's region when chunks are placed concurrently).
Such an offset is past the dataset, so it now saturates and the chunk
writes nothing, as the concurrent-placement check already assumed. No
change on 64-bit targets, where the cast cannot wrap (so no test here).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 10:37:40 -05:00
osobhandClaude Opus 5.5 94b6df986c docs: chunked full reads decode in place; small pools no longer block 
CHANGELOG entry for the chunked read changes, and the known-issues entry
on concurrent chunked reads updated: both causes it names (per-read page
faults, readers waiting on a small pool) are fixed; the 16-thread
comparison with h5py stays open until re-measured on an idle machine.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 10:36:44 -05:00
osobhandClaude Opus 5.5 6b3d003950 fix(format): refuse chunk index entries libhdf5 mis-reads 
- A dataset without filters stores every chunk at the chunk's full size.
  A chunk its index records at another size was read at that size, with
  the rest of the chunk left as zeros (cve-2025-44904's
  Scale_offset_float_data_le: 38- and 37-byte chunks for 48-byte chunks,
  where HDF5 2.0 fills the rest with whatever its buffer held). It is now
  refused, as later libhdf5 releases refuse it ("incorrect chunk size
  returned from index for unfiltered chunk"):
  chunked_read::list_chunks_for_read, used by every read path.
- A v1 B-tree chunk key carries 0 in the element-size dimension. libhdf5
  compares that coordinate when it looks a chunk up, so whether it finds a
  chunk keyed otherwise depends on where the key falls (in cve-2025-44905
  /Shuffle_float_data_le, offset 4096, it does not, and h5py reads fill
  values); we read the chunk. Such a key is now refused.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 10:36:01 -05:00
osobhandClaude Opus 5.5 d110b1d945 fix(format): scale-offset and shuffle decode chunks as libhdf5 does 
Scale-offset (filter 6) now follows H5Z__filter_scaleoffset:
- the packed codes start at byte 21 whatever size the chunk records for
  minval (libhdf5 reads min(8, size) bytes of minval and always starts the
  codes at buf_offset 21). We started them after minval plus 8 bytes, so a
  chunk recording a size of 0 (cve-2025-44905 /Scale_offset_short_data_be)
  decoded differently from h5py;
- with a fill value defined, a code equal to the all-ones code of minbits
  bits is the fill value, including minbits 0 (code 0): a chunk of nothing
  but fill values read as minval;
- minbits of the full width stores the elements as they are (no minval
  added), and an integer scale factor of the full width means the chunk was
  left untouched; minbits or a scale factor wider than the type is an
  error;
- the class parameter (integer or float) decides the decode, a scale type
  that does not match it is refused, and E-scale is refused, as in libhdf5
  (no library writes it; it was decoded here unchecked);
- minval is the stored bytes zero-extended, as libhdf5 reads it.
Codes past the end of the chunk stay an error, as in libhdf5 releases
after 2.0 ("Buffer too short"; 2.0 reads past the buffer, cve-2025-2308).

Shuffle (filter 2) uses its own parameter as the element size, as libhdf5
does, instead of the dataset's element size; a parameter larger than the
chunk leaves the chunk as it is (cve-2025-44905 /Shuffle_float_data_be),
and a parameter of 0 is an error.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 10:32:03 -05:00
osobhandClaude Opus 5.5 b3058ca46e feat: decode the superblock extension at open; read metadata cache images 
libhdf5 decodes the messages of a v2/v3 superblock's extension when it
opens a file (H5F__super_read) and refuses the file when one does not
decode. We never looked at them, so we opened cve-2020-10810 (a File
Space Info message too short for the free-space manager addresses it
announces) and cve-2020-10812 (a metadata cache image past the end of the
file), both of which libhdf5 refuses.

A file written with a metadata cache image keeps its metadata cache
entries in an image block the extension points at; libhdf5 loads them
over the file's own bytes before it reads any metadata
(H5C__load_cache_image, H5C__reconstruct_cache_contents). In
h5clear_mdc_image.h5 the root group's header exists only in the image, so
every reader failed with InvalidObjectHeaderVersion(0).

The new clawhdf5_format::superblock_ext module:
- read_superblock_extension decodes the v1 B-tree K, File Space Info and
  Metadata Cache Image messages with libhdf5's checks (versions, page size
  512 B .. 1 GiB, the addresses a persisting message lists, the image
  inside the file), with the new FormatError::InvalidSuperblockExtension;
- apply_cache_image checks an image block as libhdf5 does (signature,
  version, recorded length, entry types, rings, ages, addresses inside
  the file and not repeated, flush-dependency parents) and returns the
  file's bytes with every entry written at its address
  (FormatError::InvalidCacheImage);
- metadata_view does both.

File, MmapFile and LazyFile (and so h5rs) call metadata_view at open and
read an image file through the patched copy; the conformance probe does
the same. The image's trailing checksum is not verified, as libhdf5 does
not verify it. tests/fixtures/h5clear_mdc_image.h5 is libhdf5's own test
file.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 10:28:28 -05:00
osobhandClaude Opus 5.5 d3d73676c0 style(format): iterate the dimensions when checking them against their maxima 
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 10:28:10 -05:00
osobhandClaude Opus 5.5 56abaec75e docs: Blosc2 reads (read only); ZFP is the one plugin filter left 
Record the `blosc2` feature in the changelog, the README's feature table
and the crate table, and mark the Blosc2 half of the known "Filters"
issue fixed (dated, with the conformance run that shows h5ex_d_blosc2
reading). What stays open: ZFP, writing Blosc2, and the Blosc2 features
hdf5plugin never writes (dictionaries, lazy chunks, variable-length
blocks, user-defined codecs and registered filters), which are errors.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 10:22:52 -05:00
osobhandClaude Opus 5.5 7334e21c93 feat(format): read Blosc2 (filter 32026) in pure Rust 
hdf5plugin's Blosc2 was a clear "not implemented" error. It stores each
HDF5 chunk as a Blosc2 contiguous frame; for chunks of 2+ dimensions the
frame holds a B2ND array whose data are cut into (padded) blocks stored
one after another.

filters_blosc2 (feature `blosc2`, in `plugin-filters`; the facade
forwards both) decodes, following c-blosc2's decoder and hdf5-blosc2's
blosc2_filter.c:
- the frame: the msgpack header's fixed fields, the metalayer index, the
  offsets chunk (with the special zero/NaN/uninitialised offsets) and
  chunk lookup;
- Blosc2 chunks: 16- and 32-byte headers, special chunks (zeros, NaN,
  uninitialised, one repeated value), split and unsplit streams, zero and
  run-length streams, and the filter pipeline run backwards (shuffle,
  shuffle with a byte-group size, bit shuffle including the version-2 and
  later handling of a partial group of 8, delta against the first block,
  truncated precision);
- the codecs, shared with Blosc 1: BloscLZ, LZ4/LZ4HC, Zlib, Zstandard;
- B2ND arrays: blocks gathered into C order, padding dropped, several
  chunks per array, and the array shape checked against the chunk shape
  in cd_values as the HDF5 filter does.
Dictionaries, lazy chunks, variable-length blocks, user-defined codecs
and registered filters (e.g. bytedelta) are errors. Uninitialised chunks
read as zeros. No encoder.

Tests: h5py + hdf5plugin write every codec x filter (none, shuffle,
bitshuffle, delta) and levels 0-9 over the plugin-filter cases, then
i1..u8/f4/f8 in 1-D to 5-D chunks with partial edge chunks, datasets of
zeros, one value and NaN, and Fletcher32 before Blosc2 (plain frames for
n-D chunks); clawhdf5 reads each exactly as its unfiltered twin, and
truncated precision exactly as h5py reads it. Fixture frames from
python-blosc2 (tests/fixtures/blosc2/generate.py) cover what hdf5plugin
never writes: special chunks, delta over many blocks and odd type sizes,
odd bit-shuffle blocks, forced splitting, multi-chunk B2ND arrays with a
zero chunk, and the refused features. The decoder is fuzzed (random and
mutated frames and chunks: no panic, output within the limit).

Conformance: h5ex_d_blosc2.h5 now reads (576 of 697 ok, baseline 575).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 10:22:22 -05:00
osobhandClaude Opus 5.5 b22b15f00a fix: refuse at open the dataset storage libhdf5 refuses at open 
libhdf5 checks a dataset's storage when it opens the dataset
(H5D__contig_check, H5D__compact_init): the element count times the
element size must not overflow, contiguous storage must end inside the
file, compact data must be the dataset's size. File::dataset opened
cve-2024-32624's /Dset_OBJREF (2^62 + 2 references of 8 bytes) and
reported its shape; only reading failed.

data_read::check_dataset_storage makes those checks (new
FormatError::InvalidDatasetStorage), and File, MmapFile and LazyFile run
it whenever they open a dataset (by path, by address, from a group), as
does the conformance probe. As before, a datatype, dataspace or layout
that does not decode is left for the read to report, so such a dataset
still opens and its attributes still read. An empty contiguous dataset at
a defined address, which libhdf5 refuses, is still accepted: clawhdf5 up
to v2.7.0 wrote them.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 10:22:04 -05:00
osobhandClaude Opus 5.5 9e608b975c format: decode selected chunks into reusable buffers 
A selection read (a hyperslab or points covering at most half the
dataset) decoded each chunk it overlaps into fresh buffers, one per
filter stage; it now uses the thread's chunk-decoding scratch like the
full readers. Covered by tests/chunked_read_paths_interop.rs (small
hyperslabs and points over every filter and type) and the partial-read
equivalence tests.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 10:20:07 -05:00
osobhandClaude Opus 5.5 9e9b849dd7 read chunked datasets straight into the typed output 
read_f32/read_f64/read_i32/read_i64/read_u64 of a chunked dataset that
stores exactly that type in native byte order now decode every chunk
straight into the Vec<T> they return (data_read::read_chunked_native),
on File (through its chunk cache), MmapFile and LazyFile. Before, the
chunks went into a byte buffer that read_as_* then copied into a second,
typed one: two dataset-sized allocations and a full extra copy per read.
The output is zeroed pages from the allocator, backed by transparent
huge pages when large, like the byte reader's. Other types and byte
orders, and datasets with no storage or external data, keep converting
through the byte readers; unallocated chunks read as the fill value as
before.

tests/chunked_read_paths_interop.rs checks every chunked read path
(File twice, so cached; from_bytes; MmapFile; LazyFile; small, strided
and point selections; with and without the parallel feature) against
h5py for 1-8 byte integers and 2-8 byte floats in both byte orders,
through deflate, shuffle, Fletcher32, LZF, SZIP and Blosc, with partial
edge chunks, sparse datasets with default and non-default fill values,
and datasets larger than the chunk cache. A filter this build lacks must
be an error (or, when an optional filter declined every chunk, the right
data).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 10:18:29 -05:00
osobhandClaude Opus 5.5 193a5f8a82 writer: track attribute creation order with track_order 
h5py's track_order=True orders attributes as well as links; the writer
tracked links only. A tracking object's header now sets the attribute
creation order tracked/indexed flags and carries per-message creation
orders, an Attribute Info message holds the next order (inline too),
and dense storage gets a type-9 creation-order index. The file default
applies to datasets, with DatasetBuilder::track_order per dataset; more
than 65 535 attributes on a tracking object is an error (libhdf5's
counter is 2 bytes). The reader lists such attributes in creation
order.

h5py lists them in order (inline, dense, 20 000 on one dataset) and
keeps numbering in r+ mode, including its inline-to-dense move.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 10:17:37 -05:00
osobhandClaude Opus 5.5 3aab433edb fix(format): dataspaces and contiguous storage as libhdf5 reads them 
- A simple dataspace of rank 0 holds one element in libhdf5 (the product
  of no dimensions; h5py reads it as shape ()). num_elements() said 0, so
  cve-2020-18494's /dset1 failed with DataSizeMismatch { expected: 0 }.
- A contiguous dataset whose storage is larger than its elements reads:
  libhdf5 reads the elements from the start of the storage and ignores the
  rest (H5D__contig_check checks only that they fit in the file). We
  required the sizes to be equal, so the scalar /Dset1 of cve-2024-32623
  and cve-2025-2309 (240 bytes of storage for one int) failed. Storage too
  small for the elements is still an error. data_read::contiguous_read_len
  is the rule, used by every contiguous read path.
- Dataspace::parse refuses what H5O__sdspace_decode refuses: more than 32
  dimensions, a rank on a scalar or null dataspace, a dimension larger
  than its maximum (new FormatError::InvalidDataspace).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 10:17:15 -05:00
osobhandClaude Opus 5.5 1c1af460b6 docs: design for range reads (S3/HTTP, wasm lazy loading, SWMR, 32-bit) 
Inventory of the 109 whole-file parser functions, a measurement of the
metadata ranges clawhdf5 touches on three corpus files against libhdf5,
options (storage trait, virtual slice, metadata prefetch, userfaultfd)
with how ros3, h5py+fsspec, pyfive, h5wasm and object_store do it, and
an incremental plan: M0 name-index lookups, M1 metadata over a Storage
trait, M2 raw data with batched ranges, M3 object_store backend,
M4 wasm fetch-driven lazy loading.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 10:17:02 -05:00
osobhandClaude Opus 5.5 c5cd14c2b2 docs: tools to measure how a range reader would read HDF5 
inventory.py counts the functions and call sites that take the whole
file as &[u8]; range-trace (standalone crate, x86-64 Linux) records every
load clawhdf5 makes from a file by mprotect + single-step, unchanged
library code; libhdf5_reads.py counts libhdf5's reads through h5py's
fileobj driver and prints a dataset's chunk extents.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 10:17:02 -05:00
osobhandClaude Opus 5.5 16b7359485 fix(format): a v1 group with an empty link name fails its listing 
libhdf5 refuses to list a symbol-table group that has an entry with an
empty name (H5G__ent_to_link: "invalid link name"), so h5py cannot list
cve-2021-46244's /BAG_root. We listed it, with an object at "/BAG_root/"
(the empty name, pointing at address 0). resolve_v1_group_entries — the
listing — now fails with the new FormatError::InvalidLinkName; path
lookups still find the group's other names, as libhdf5's by-name lookup
does.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 10:15:50 -05:00
osobhandClaude Opus 5.5 1207df5189 feat(format): ObjectHeader::object_class, libhdf5's object classification 
libhdf5 decides what an object header is in a fixed order
(H5O__obj_class_real): a group if it has a Symbol Table or Link Info
message, a dataset if it has a Datatype *and* a Dataspace message, a named
datatype if it has a Datatype message. The conformance probe called any
header with a Data Layout message a dataset, so cve-2024-33874's /Dset1 (a
datatype and a layout, no dataspace), which h5py opens as a named
datatype, was reported as a dataset we failed to read
(MissingMessage(Dataspace)). The probe now classifies with
object_class().

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 10:14:31 -05:00
osobhandClaude Opus 5.5 f0db817678 format: decode full chunked reads straight into the output 
Full reads of a chunked dataset (the cached reader behind the facade's
read_* and the uncached one behind mmap/lazy files and verify_provenance)
now decode each chunk into this thread's reusable scratch buffers and copy
it straight to its place in the output. Before, the cached reader decoded
batches of 128 chunks into fresh Vecs and the uncached one decoded every
chunk of the dataset into its own buffer before assembling any: a new
256 KiB allocation (and its page faults) per chunk and per filter stage.
Chunks are still inserted into the file's chunk cache when the whole
dataset fits in it.

With the parallel feature the calling thread now decodes too, sharing the
chunks with whichever rayon workers are free (run_with_helpers): a helper
the busy pool only starts after the read is done returns at once. Before,
the caller handed every chunk to the pool and slept, so readers outside a
small pool (2-4 threads) queued behind its workers; with a one-thread pool
the reads went sequential. Chunks are placed concurrently only when the
index puts them on the chunk grid at distinct places (a corrupt index is
read one chunk at a time), and the error returned is still the first
failing chunk's.

Fix: a chunk stored unfiltered in a filtered dataset (every filter-mask
bit set) that is shorter than a chunk read as zeros where its data was
missing through the cached reader (the facade's read_*); it is now an
error naming the chunk, as the uncached reader already made it.

Regression tests, both failing before this change:
tests/busy_decode_pool.rs (both workers of a two-thread pool busy, four
readers) and short_unfiltered_chunk_of_a_filtered_dataset_is_an_error.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 10:13:31 -05:00
osobhandClaude Opus 5.5 9e59499c56 conformance: fix three reference-probe artefacts 
ref.py and compare.py reported 13 files as mismatches or our-errors that
were artefacts of the harness, not differences between the readers:

- User-defined links (tall.h5, tudlink.h5, twithub*.h5, tmany.h5, ...):
  h5py's `get(name, getlink=True)` reports a user-defined link as a
  HardLink, so ref.py listed it as an object. Read the link type from
  H5Lget_info instead.
- Objects h5py cannot open (cve-2019-8397/8398, cve-2021-46243,
  cve-2024-32618): the probe deduplicates by header address, ref.py by
  ObjectID, which an unopenable object does not have, so each extra hard
  link to it was listed again. Deduplicate those by link address.
- Nested array types (tarray3.h5): h5py expands them into trailing dims;
  hash_values stripped one level and numpy broadcast every element into a
  whole subarray. Strip every level.

compare.py no longer compares the attributes or links of an object h5py
could not open at all (cve-2018-17438/17439, cve-2019-9151): h5py read
none, so ours are neither extra nor errors against it.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 10:13:11 -05:00
osobhandClaude Opus 5.5 d63c76e7ab writer: v2 B-trees with internal nodes (no 65 535-record limit) 
Dense link and attribute indexes and the chunk index for several
unlimited dimensions were single leaves, capping them at 65 535
records. btree_v2_write builds trees of any depth, with node capacities
and pointer widths from libhdf5's H5B2__hdr_init arithmetic (now shared
with the reader as btree_v2::node_info) and libhdf5's node sizes (512
dense, 2048 chunks). Indexes that fit the old one-leaf layout are
written byte for byte as before (compared for 10..65 535 links, attrs
and chunks, tracked and filtered).

Tests: 100 000 links (short names; long names with creation order),
70 000 attributes, 200 000 chunks (and 80 000 deflated), read by h5py,
h5dump and clawhdf5 and edited by h5py r+; h5rs check on the same
shapes, asserting depths 2-3.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 10:12:07 -05:00
osobhandClaude Opus 5.5 cc1c872a93 format: decode chunks into reusable scratch buffers 
decompress_chunk_exact_with decodes a chunk into a DecodeScratch the
caller keeps between chunks, instead of a new Vec per chunk and per filter
stage. Deflate inflates into a kept buffer with a reset (not rebuilt)
inflater, shuffle interleaves into the other buffer, and Fletcher32 checks
and drops its checksum in place (on the stored bytes when it is the first
filter undone). Other filters go through the registry as before. Output
and errors are those of decompress_chunk_exact; a unit test checks that
for every pipeline shape and filter mask with one reused scratch.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 10:05:31 -05:00
osobhandClaude Opus 5.5 7acfb79584 writer: order dense name indexes by hash, then name 
libhdf5 compares the name when two hashes are equal; the writer broke
ties by insertion order, and libhdf5 could not find one of two names
whose lookup3 hashes collide (k69209 / k155448). Test fails before the
fix with h5py's KeyError.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 10:03:22 -05:00
osobh de2a53f613 Merge pull request 'Fast contiguous and concurrent reads, VL data, nested groups and links, Python bindings' (#15) from feat/p2-perf-coverage into main
CI / test-arm64 (push) Successful in 1m24s
Details
CI / test (push) Successful in 12m27s
Details 
Reviewed-on: #15
 2026-09-26 14:57:00 +00:00
osobhandClaude Opus 5.5 dda28d6c72 bench: concurrent reads re-measured after the read fixes
CI / test-arm64 (pull_request) Successful in 1m34s
Details
CI / test (pull_request) Successful in 7m30s
Details 
Idle tank at 408f69e, h5py re-run in the same session. Contiguous reads
went from 0.25x to 1.44x h5py (full) and 0.12x to 6.3x (256x256
hyperslabs) on one thread; deflate full reads at 8 threads 887 -> 2943
MB/s (h5py processes 3042). Full chunked reads at 16 threads are still
0.69x-0.76x h5py processes; the issue stays open.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 09:28:51 -05:00
osobhandClaude Opus 5.5 408f69ec1d docs: conformance report after the perf and coverage merges (575 of 697 ok) 
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 09:18:54 -05:00
osobh 73a01f1256 Merge branch 'feat/p2-python-bindings' into feat/p2-perf-coverage 
# Conflicts:
#	CHANGELOG.md
#	README.md
 2026-09-26 09:10:57 -05:00
osobh 956e55c76a Merge branch 'feat/p2-writer-groups-links' into feat/p2-perf-coverage 
# Conflicts:
#	CHANGELOG.md
#	crates/clawhdf5-tools/tests/h5rs_interop.rs
 2026-09-26 09:10:50 -05:00
osobh 846c35455d Merge branch 'feat/p2-vl-strings' into feat/p2-perf-coverage 
# Conflicts:
#	CHANGELOG.md
 2026-09-26 09:10:35 -05:00
osobh ca779b2864 Merge branch 'perf/p2-contiguous-reads' into feat/p2-perf-coverage 
# Conflicts:
#	CHANGELOG.md
#	docs/known-issues.md
 2026-09-26 09:10:26 -05:00
osobh 20bd381c87 Merge branch 'perf/p2-chunk-cache-scaling' into feat/p2-perf-coverage 2026-09-26 09:10:17 -05:00
osobhandClaude Opus 5.5 5a202f3791 fix(format): a VL element at the undefined heap address is an error 
libhdf5 fails to read a VL element whose global heap address is
undefined (all 0xff), even at length 0 ("addr undefined"); we returned
"" (or an empty sequence) in every reader. Checked with h5py first:
libhdf5 writes a null element with address 0, which still reads as
empty, and h5py writes "" as a zero-size heap object at a real address,
so no file they write relies on the old behaviour. read_vl_bytes now
treats address 0 as null whatever the length, as VlResolver does.

Tests, each failing before: vl_data unit test (8- and 4-byte offsets,
lengths 0 and 1); clawhdf5 vl_data_interop
a_vl_element_at_the_undefined_heap_address_fails_like_h5py (also checks
where h5py writes ""); h5rs dump --json and check --data on the patched
`undef` dataset; clawhdf5-wasm vl_strings.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 09:06:46 -05:00
osobhandClaude Opus 5.5 8dcce084ca test: the v4 chunk-index selection test passes clippy -D warnings 
A type alias for the hyperslab tuple, and as_chunks for the i32 decode.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 09:05:31 -05:00
osobhandClaude Opus 5.5 45d617c39e docs: say when a selection read decodes more than the selection 
The READMEs said ds[...] reads only the selected elements, and the
facade's read_selection docs that only intersecting chunks are
decompressed. The bounding-box path runs only when the box covers at
most half the dataset; larger boxes (any strided slice across the
dataset), compact, virtual and unwritten datasets and chunked ones with
a non-default fill value decode the whole dataset. The READMEs, the
facade and format docs, the bindings' docstrings and known-issues now
say so, and how index lists are read.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 09:04:48 -05:00
osobhandClaude Opus 5.5 d345ffbf80 fix(tools,wasm): resolve VL data through the library's VlResolver 
h5rs (dump, ls, diff, check --data) kept its own lenient VL decoder:
a heap object longer than its element was cut to the element's length
(libhdf5 and h5py refuse it), a null string printed "" where h5dump
prints NULL, the stored element size was trusted, and every heap
collection was kept as an owned copy for the whole run. It now resolves
each element with VlResolver::element / string_element (new: one element
in place, borrowing from the file), and refuses a VL type whose stored
element size is not 4 + offset size + 4, as File does. H5::heap_object
and its cache are gone. h5diff compares a null VL string equal to an
empty one; so does h5rs diff.

clawhdf5-wasm already resolved VL strings with read_vl_strings; it now
uses VlResolver and checks the stored element size before reading, as
File::read_string does.

Tests (h5py writes the files, patched for "a\0b", a null element and
mis-sized heap objects, with 8- and 4-byte offsets):
- h5rs_interop dump_prints_vl_data_like_h5dump: byte-identical to h5dump;
- dump_json_vl_values_match_h5py: h5py's values, errors where h5py fails;
- check_data_flags_mis_sized_vl_heap_objects;
- clawhdf5-wasm tests/vl_strings.rs: wasm, File and h5py agree.
All four fail before. check --data over the 150 cve_hdf5 CVE and fuzzer
files now passes 15 (h5dump rejects 8 of them), was 16 and 9: the
stored-size check flags cve-2024-32608. h5rs-check-ok-files.sh --data:
0 of 422 flagged; h5rs-fuzz.sh: clean on 180 files.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 09:04:21 -05:00
osobhandClaude Opus 5.5 17edfe2cf0 test(py): detect a held GIL, and errors h5py does not raise 
test_threads_read_the_same_file passed with the GIL held. The new
test_reads_release_the_gil measures the longest stall of a spinning
Python thread while another reads: with py.detach removed from the read
it stalled 0.062 s of a 0.064 s read and failed; with it, about 3 ms.
test_errors_match_h5py now compares the result whenever h5py reads the
key, instead of only checking that we raise when h5py raises, over a
longer key list.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 09:03:53 -05:00
osobhandClaude Opus 5.5 546fdb84fa docs: dense storage fixes and the real limits of big groups 
The changelog, known issues and README said a group holds up to 65 535
links while a group of about 17 000 was already unreadable. Record the
fixes (child indirect blocks, the next-block offset, the index leaf cap,
refusing oversized dense messages, hard-link memoisation, dataset
attribute overwrite) and the limits that remain true: 65 535 links or
dense attributes per object, and 65 515 bytes per dense message.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 09:02:47 -05:00
osobhandClaude Opus 5.5 05b0192a60 fix(py): a 0-d integer array indexes like an int 
ds[np.array(1)] went down the index-list path, where tolist() returns a
scalar and extracting a list of indices raised a confusing TypeError.
h5py treats it as an integer index; so do we now. The h5py comparison
keys include 0-d arrays (signed and unsigned) on each axis; they failed
before.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 09:02:28 -05:00
osobhandClaude Opus 5.5 8bcae3c78e fix(format): a dataset attribute set again replaces the earlier value 
b0a1e4f fixed this for group and root attributes only. Setting a dataset
attribute twice still wrote two attribute messages with one name, and h5py
read back the first value: set_attr("a", 1) then set_attr("a", 2) read as
1, and list(attrs) was ["a", "a"]. DatasetBuilder::set_attr now replaces
the earlier value, compact or dense. Likewise, a hand-set attribute named
like a provenance attribute (_provenance_sha256, ...) is replaced by the
computed one instead of being written next to it and read first.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 09:02:09 -05:00
osobhandClaude Opus 5.5 f0ecae38b6 perf(py): datasets and groups keep their address; groups their links 
Every ds[...] and g[k] resolved the path from the root again, two or
three times per open, and resolving a name in a large group scans its
links: visiting a group was O(n^2). 4000 scalar datasets in one group
took 39 s (v1 group) and 131 s (dense) to list, read and re-read; now
0.3 s each. A Dataset keeps its object address, a Group (and the file's
root) its address and, after the first lookup, its link table.

New facade API File::dataset_at(address), tested in integration_tests.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 09:01:54 -05:00
osobhandClaude Opus 5.5 400e3a9fec fix(format): resolve each hard link once 
A hard link's target may go through other hard links, and each was
resolved again every time a path went through it. With each link's
target naming the previous link twice (g/s{i} -> /g/s{i-1}/s{i-1}) the
work doubled per link: finish() took 46 s for 26 links in a debug build,
and 60 would never finish. Resolved links are now remembered, so the work
is linear in the links, and a hard link met again while it is being
resolved is reported as a cycle by name. The depth limit (64) still bounds
the recursion through links not yet resolved.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 09:01:13 -05:00
osobhandClaude Opus 5.5 bd1d8f1a59 fix(format): keep a dense index leaf within 65 535 records 
The link and attribute name indexes are one v2 B-tree leaf, sized to the
next power of two. libhdf5 takes a leaf's capacity from that node size,
but a leaf's record count is a 2-byte field. From about 47 700 links the
node had room for more than 65 535 records, so adding a link in h5py
overflowed the count: a group of 65 535 links crashed h5py, or could no
longer be listed ("unknown link class"). The node is now capped at a full
leaf of 65 535 records, so libhdf5 splits it instead.

Dense attributes now go through the same index builder. Their record
count was written modulo 65 536, without error; more than 65 535
attributes on one object are now refused, like links.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 08:58:49 -05:00
osobhandClaude Opus 5.5 751edeb7e6 fix(format): refuse a dense link or attribute too big for the heap 
A message in dense storage is a fractal heap object, and an object must
fit one direct block: 65 515 bytes here, since the writer has no
huge-object path. A bigger one (a soft link with a 80 000-byte target in
a group of more than 8 links) was written without error, cut off at the
end of its block, and libhdf5 could not list the group ("object overruns
end of direct block"). finish() now fails with an error that names the
limit, for links and for dense attributes; a 65 001-byte soft link target
still works and h5py reads it back. The heap packer also skips a child
indirect block whose blocks are all too small for the next object instead
of walking it.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 08:57:52 -05:00
osobhandClaude Opus 5.5 b43bd2e67f perf(py): read an index list one group of chunks at a time 
Each run of consecutive indices was its own uncached hyperslab read, so
a list over a compressed chunked dataset decoded the same chunk once per
run (d[range(0, 200000, 40)] over 20 gzip chunks: 8 s, h5py 0.014 s).
Plan::reads now groups the indices — a group ends only where a whole
chunk holds no selected index, or, unchunked, at a gap over 64 KiB — and
the selected rows are gathered from each group's block in Rust. Now
3.8 ms (h5py 4.1 ms, release, tank). The new test (1-D, 2-D and
contiguous, compared with h5py, 2 s bound) took 5.8 s before.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 08:57:16 -05:00
osobhandClaude Opus 5.5 81a0e8685d fix(format): write child indirect blocks in big fractal heaps 
Dense link and attribute storage keeps its messages in a fractal heap. Its
root indirect block holds direct blocks up to 64 KiB, 512 KiB in all; rows
past that are child indirect blocks. The writer kept adding rows of direct
blocks instead, and libhdf5 and h5rs read them as indirect blocks: a group
with 20 000 links of 20-byte names was written without error and could not
be listed ("incorrect metadata checksum"), and 150 dense attributes of up
to 56 KB could not be opened. The heap writer now follows the doubling
table: rows past the direct ones hold child indirect blocks, each with its
own rows, nested as deep as the heap needs.

Two more heap bugs are fixed on the way. An object bigger than the next
block's free space was written into it anyway and cut off; the block is
now left unallocated and the object goes in the first block big enough, as
libhdf5 skips blocks. And the header's next-block offset was 0, so libhdf5
adding a link to such a group overwrote the heap's first block ("bad
version number for message"); it is now the offset after the last block.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 08:57:01 -05:00
osobhandClaude Opus 5.5 41b7837d0a fix(format): bound what a VL read retains on a crafted global heap 
VlResolver kept an owned copy of every object of every heap collection
it parsed, for the whole read. Collections nested inside each other's
object data, 32 bytes apart with each element pointing at a different
one, made retained memory O(elements x file size): 1.58 GB for a 744 KB
file (read_vl_strings did the same before VlResolver). Chaining every
collection's objects into one shared run of tiny objects made parse
time O(elements x objects) as well. libhdf5 refuses these files.

- The cache records where each object lies (GlobalHeapCollection::
  parse_index, new) instead of copying it, and is dropped past a 32 MiB
  budget.
- A collection overlapping one already read is an error: libhdf5 gives
  every collection its own block, so only a crafted file has them.
- parse and parse_index refuse a collection that runs past the end of
  the file and an object that runs past the end of its collection.

tests/vl_heap_bounds.rs measures peak heap use with a counting
allocator: 129 MB and 350 MB live before on its two crafted files (64 KB
and 176 KB), 97 KB and 0.9 MB now. Conformance unchanged at 575 of 697.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 08:56:42 -05:00
osobhandClaude Opus 5.5 8c51b05b9c fix(py): index lists of padded compounds no longer return uninitialised padding 
np.concatenate copies structured dtypes field by field into np.empty, so
the padding of ds[[0, 3, 6]] held process memory. The runs' bytes are
joined in Rust, whole elements at a time, before anything becomes numpy:
the padding is the file's bytes (h5py's) and the result is still a view
of the Rust buffer. The h5py comparisons now compare every byte of
structured values; the new test failed on the padding before.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 08:54:04 -05:00
osobhandClaude Opus 5.5 24412a0e59 fix(py): a panic in the library raises clawhdf5.InternalError, not PanicException 
PanicException derives from BaseException, so `except Exception` let a
library bug through. Every call from the bindings into the library now
runs under catch_unwind and a panic becomes InternalError (RuntimeError)
naming the object. Tests: a hidden hook panics inside the guard; and the
v4 chunk indexes are compared with h5py from Python — with the library
fix reverted, ds[0:30] of the implicit-index dataset now raises
InternalError instead of aborting the test run.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 08:52:55 -05:00
osobhandClaude Opus 5.5 3bcd443e63 fix(format): selections of v4 implicit-index chunked data no longer panic 
read_raw_data_selection's chunked fallback (taken when partial_read
declines, e.g. a bounding box over half the dataset) handed the layout's
chunk dimensions, element-size dimension included, to
generate_implicit_chunks, which indexed past the dataset rank. It then
decoded the whole dataset regardless, so the enumeration is gone: the
arm decodes and extracts for every chunk index.

The new test reads small and large hyperslabs of all five v4 indexes
written by h5py and compares with h5py's values; it panicked before.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 08:49:54 -05:00
osobhandClaude Opus 5.5 37770f594a docs: the rayon fix covers a one-thread pool, not the h5py-process gap 
The review measured the default pool unchanged (about 2900 MB/s at 16
threads before and after) and still short of 16 h5py processes; small
pools still make outside readers wait. Say so instead of marking the
scaling issue fixed.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 08:48:05 -05:00
osobhandClaude Opus 5.5 a5e41c1a53 fix(read): decode on the calling thread when rayon's pool has one thread 
Full reads of chunked datasets handed their chunks to rayon. With a
one-thread pool (concurrent_read --decode-threads 1, RAYON_NUM_THREADS=1)
every thread reading through a File queued behind that single worker, so
16 readers decoded on one core: per-thread CPU time showed one thread
doing all the decoding and the readers almost none, and full reads
stopped at about 2x one thread. The cached full-read path and the
uncached reader behind verify_provenance now decode inline when the pool
cannot parallelise (parallel_read::pool_can_parallelise).

The File's chunk cache was the suspect but not the cause: datasets over
its budget were already read without inserting, and skipping its lookups
gained only a few percent at 16 threads.

The regression test keeps a one-thread global pool's worker busy and
requires a full read and verify_provenance to finish anyway; before the
fix both waited for the worker (timed out).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 08:40:10 -05:00
osobhandClaude Opus 5.5 b0a1e4f9a6 fix(format): a group attribute set again replaces the earlier value 
Setting a group or root attribute twice wrote two attribute messages with
the same name, and h5py read back the first value: set_attr("w", 1) then
set_attr("w", "two") read as 1. The later value now replaces the earlier
one, as `attrs[name] = v` does in h5py, including when a group is merged
from two builders.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 08:35:16 -05:00
osobhandClaude Opus 5.5 bd36fe883b fix(format): flag non-ASCII link names as UTF-8 
The writer marked every link name ASCII, so a name such as "größe" was
stored as UTF-8 bytes under the ASCII character set (h5py reports cset 0
for it). Names that are not plain ASCII now carry the UTF-8 flag, as h5py
writes them; ASCII names are unchanged.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 08:34:04 -05:00
osobhandClaude Opus 5.5 d102c06306 feat(format): nested groups, soft/hard/external links and creation order in the writer 
FileWriter wrote the root group plus one level of groups, and refused
path-like names. The writer now flattens its builders into a group tree
(writer_tree.rs) before layout:

- A name may be a path ("a/b/x", "/a/b/x" at the root); missing
  intermediate groups are created as h5py does, and GroupBuilder gains
  create_group/add_group so builders nest to any depth. A group added at a
  path that already holds a group is merged into it (require_group);
  any other repeated name, an empty or "." component, or an absolute path
  below the root is an error.
- add_soft_link, add_hard_link and add_external_link on FileWriter,
  FileBuilder and GroupBuilder. Hard-link targets are resolved to objects
  at finish (through other hard links; a missing target, a soft link on the
  way or a cycle of paths is an error). Objects with several hard links get
  an Object Reference Count message so libhdf5 can delete one link without
  freeing the object.
- track_order(true) per group, or as the file default, tracks and indexes
  link creation order: Link Info flags and max order, the order in each
  Link message, and a type-6 creation-order B-tree for dense groups.
- A group's link index is one B-tree leaf; more than 65535 links is an
  error.

Groups are laid out depth-first from the root, datasets group by group,
and untracked groups keep writing datasets, then groups, then other links:
files with one level of groups are byte-identical to before.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 08:33:46 -05:00
osobhandClaude Opus 5.5 6e8421a81e build: record libc in the conformance probe's lockfile 
clawhdf5-format now depends on libc on Linux (huge-page advice for read
buffers); the probe's committed lockfile picks that up.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 08:27:09 -05:00
osobhandClaude Opus 5.5 8ce6eca34d feat(facade): read VL strings and VL sequences through File 
VL-string datasets (h5py's default str dtype) failed read_string with
"type mismatch: expected String, got VariableLength". read_string now
reads fixed- and variable-length strings, with h5py's values (a string
ends at a NUL, a null element is ""). New:
- Dataset::read_string_bytes: each VL string's exact bytes;
- Dataset::read_string_selection: hyperslabs/points of either kind;
- Dataset::read_vlen::<T>() and read_vlen_selection::<T>(): VL sequences
  of numbers as Vec<Vec<T>>, T in f64/f32/i64/i32/u64, converted like the
  other typed readers;
- File::decode_strings / decode_string_bytes / decode_vlen: VL values in
  compound fields and AttrValue::Raw attributes;
- MmapDataset and LazyDataset: read_string for VL strings,
  read_string_bytes and read_vlen.

tests/vl_data_interop.rs checks every path against h5py with 8- and
4-byte offsets: scalar, 1-D and 2-D, ASCII and UTF-8, empty strings,
contiguous, compact, chunked with gzip and shuffle, unwritten and partly
written chunks, hyperslabs, compound members, attributes, a big-endian
base type, and a patched file with an embedded NUL and mis-sized heap
objects. NetCDF-4 string variables read too (netCDF4-python test).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 08:25:05 -05:00
osobhandClaude Opus 5.5 c3850a0b66 docs: the Python package — install with maturin, h5py-style reading 
README gains a Python section (maturin develop into a venv, a reading
example that was run against an h5py-written file, the supported types
and keys, what writing covers). The crate README says the same in more
detail. QUICKSTART showed clawhdf5.open()/read_f64(), which never
existed; it now shows File(...)[...].

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 08:21:48 -05:00
osobhandClaude Opus 5.5 2bc4cb46a6 perf: copy contiguous hyperslab and point reads run by run 
A 256 x 256 hyperslab of a contiguous f32 dataset read at an eighth of
h5py's speed: partial_read copied the bounding box out of the file, the
extractor then walked it element by element (a recursive call and two
bounds checks per element) into a second buffer, and read_f32_selection
converted that into a third.

Selections of contiguous data are now copied straight from the file, one
memcpy per run of elements contiguous in the file (gather.rs: a block
along the last dimension, touching blocks as one range, whole rows
merged), with no zero-filled intermediate and no full copy for large
selections. The typed selection readers copy into their Vec<T> directly
when the dataset stores T natively (new data_read::read_selection_native
and sealed NativeElement trait, which the read_as_* fast paths now share;
read_as_u64 gains one) and convert as before otherwise. The general
extractor used by the chunked paths runs on the same run walker, keeping
its old handling of unvalidated selections.

Checked against h5py (contiguous_read_interop.rs) for strided, blocked,
adjacent-block and whole-row hyperslabs, points and empty selections of
every 1-8-byte type in both byte orders, ranks 1-4.

Also keeps the huge-page threshold constant out of no_std builds, where
it was unused.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 08:21:47 -05:00
osobhandClaude Opus 5.5 f7d88bb4fb ci: build the Python package with maturin and run its tests against h5py 
ci-test.sh gains a step that lints clawhdf5-py, builds its wheel with
maturin, unpacks it under target/ (the interpreter's environment is not
touched) and runs the pytest suite, which compares reads with h5py. It
skips without maturin/pytest, and fails instead under
CLAWHDF5_REQUIRE_INTEROP=1. The CI interop venv installs maturin and
pytest, so CI runs it.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 08:21:14 -05:00
osobhandClaude Opus 5.5 f99587c27d fix(format): resolve VL elements as libhdf5 does 
Checked with h5py on a patched file:
- a VL string with an embedded NUL reads up to the NUL (libhdf5 converts
  VL strings to C strings); read_vl_strings returned "a\0b";
- an element whose global heap object is not length x base size bytes is
  an error ("Expected global heap object size does not match"); we
  returned the object cut to the length;
- a heap address of 0 is a null element whatever its length.

vl_data::VlResolver does this, caching each parsed heap collection:
read_vl_strings parsed the whole collection again for every element.
read_vl_strings and read_vl_bytes use it; check_element_size refuses a VL
type whose stored element size is not 4 + offset size + 4. The
conformance probe resolves VL values through VlResolver instead of its
own lenient copy (575 of 697, unchanged).

The new unit tests fail against the old read_vl_strings.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 08:20:19 -05:00
osobhandClaude Opus 5.5 2d4b211523 feat(py): h5py-style reads of only the selected elements, GIL released 
ds[key] read the whole dataset and sliced it in numpy, and knew six
dtypes. Keys (ints, positive-step slices, Ellipsis, one increasing index
list, compound field names) now map onto hyperslab selections, and the
facade's read_selection bytes become the numpy buffer without a copy
(PyArray::from_vec viewed as the dtype). dtype mapping follows h5py for
all integer/IEEE float widths and byte orders, bool, enum, complex, fixed
and variable-length strings, vlen sequences, opaque, array types and
(nested, padded) compounds; anything it cannot describe exactly is a
TypeError. Attributes return what h5py returns; groups and files gain
the rest of the h5py mapping interface. Reads run under py.detach.

tests/test_read_vs_h5py.py compares >500 reads with h5py 3.16 on an
h5py-written file, checks errors match, that a damaged chunk outside the
selection is never touched, and 8 threads reading at once.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 08:19:48 -05:00
osobhandClaude Opus 5.5 10da8f0d09 fix(format): read VL values in files with 4-byte offsets 
In a file with sizeof_addr = 4, a VL string attribute came back as
AttrValue::Raw, a compound's VL member failed with
GlobalHeapObjectNotFound and VL datasets failed with a size mismatch.

Two bugs: Datatype::type_size() said 16 for every VL type, while the
element is 4 + offset size + 4 bytes (12 here); and the global heap was
parsed without the padding libhdf5 puts after its collection and object
headers (both round up to 8), so with 4-byte lengths every object was
looked up 4 bytes early. Datatype::VariableLength now carries the size
its datatype message stores, and writes it back.

Checked against h5py in tests/vl_offset4_interop.rs (fails with either
fix reverted). Conformance unchanged at 575 of 697; in cve-2024-32608 a
VL attribute whose datatype claims 524304-byte elements is now an error
(h5py cannot iterate those attributes at all).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 08:19:13 -05:00
osobh 83cda847dd Merge pull request 'h5rs tools, browser reader, libhdf5 header checks, plugin filters, concurrency benchmark' (#14) from feat/p1-proof into main
CI / test-arm64 (push) Successful in 1m9s
Details
CI / test (push) Successful in 7m51s
Details 
Reviewed-on: #14
 2026-09-26 13:14:39 +00:00
osobhandClaude Opus 5.5 78c769f179 perf(format): back large read buffers with transparent huge pages 
A full read of a contiguous dataset is one memcpy from the mapped file,
yet ran at a quarter of h5py's speed on one thread: the fresh output Vec
took a page fault and a kernel page clear for every 4 KiB page written,
16384 per 64 MiB, costing several times the copy (the benchmark spent
6.2 s of 8 s in the kernel, 4.3M minor faults). numpy, so h5py, madvises
MADV_HUGEPAGE on allocations of 4 MiB or more; the typed readers' output,
the raw contiguous read and the chunk assembly buffer now do the same
(Linux only, libc as a Linux-only dependency; no-op otherwise).

New h5py comparison tests cover full and selection reads of contiguous
data for every 1-8-byte integer and float type, both byte orders, ranks
1-4, empty selections, and datasets past the 4 MiB threshold.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 08:12:55 -05:00
osobhandClaude Opus 5.5 006bf3b131 fix(py): one name, clawhdf5, for the Python distribution and module 
pyproject.toml named the distribution rustyhdf5 while the extension
module is clawhdf5, and the package's tests imported rustyhdf5, so
pytest failed at collection. Distribution, module-name and tests now
agree; the module gains __version__. maturin develop + pytest: 28 pass.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 08:09:54 -05:00
osobhandClaude Opus 5.5 8cbbef3fae fix(format): write a Group Info message in every group 
libhdf5 reads a group's Group Info message before it inserts a link, and
FileWriter wrote none, so h5py in "r+" mode could not add a link to any
group we wrote: "Unable to create link (message type not found)". Each
group header now carries a version 0 Group Info message with the default
link-phase thresholds, as libhdf5 writes for a new group.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 08:08:31 -05:00
osobh 55309dd242 Merge pull request 'Read HDF5 1.6-era files, user blocks, VDS, dense attributes and large groups' (#13) from fix/p1-read-gaps into main
CI / test-arm64 (push) Successful in 1m8s
Details
CI / test (push) Successful in 5m11s
Details 
Reviewed-on: #13
 2026-09-26 09:42:10 +00:00
osobhandClaude Opus 5.5 63648c7000 bench: concurrent-read results on tank, including where we lose
CI / test-arm64 (pull_request) Successful in 1m9s
Details
CI / test (pull_request) Successful in 7m9s
Details 
h5py threads stay flat (global lock); clawhdf5 hyperslab reads of deflate
data scale to 1244 MB/s at 16 threads (9.7x h5py threads, 0.89x h5py
processes). Two deficits recorded as open issues: full chunked reads stop
scaling at ~4 threads (chunk cache suspected), and contiguous reads are
4x (full) to 8x (hyperslab) slower than h5py single-threaded.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 02:02:17 -05:00
osobhandClaude Opus 5.5 91644d8aaf docs: conformance report with header checks and plugin filters (575 of 697 ok) 
Regenerated on tank: ok 569 -> 575 (h5ex_d_blosc, h5ex_d_bshuf,
h5ex_d_bzip2, h5ex_d_lzf; h5clear_fsm_persist_less, h5stat_err_refcount),
our-error 14 -> 10, mismatch 22 -> 20, no panics, hangs, crashes or OOM.
Baseline raised.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 01:50:02 -05:00
osobhandClaude Opus 5.5 72306c6013 fix(tools): h5rs check says why the library refused a file 
With the header checks merged, the library refuses truncated files and
misaligned chunk-index keys itself, so check reported only "file cannot
be opened" for a truncated file. It now reports the truncation (stored
end of file vs file length) or the library's error, and the misaligned
chunk test accepts the library's refusal of the key.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 01:46:48 -05:00
osobhandClaude Opus 5.5 e60bde3579 docs: h5rs check CVE counts measured with the header checks merged 
Measured on tank on the 150 cvefiles/ and fuzzerfiles/ of cve_hdf5 (the
earlier text said 180): check --data passes 16 (was 28), and h5dump
1.14.6 rejects 9 of those (was 21). It still flags none of the 418
conformance files both readers read in full.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 01:41:39 -05:00
osobhandClaude Opus 5.5 c85a8222cc test: compile the facade's parallel tests, and build them in CI 
parallel_integration.rs declared `_sequential` and used `sequential`
under the parallel feature, which nothing in CI enabled for the facade.
ci-test.sh now lints and tests the facade with parallel on.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 01:38:38 -05:00
osobh 2b68791f6a Merge branch 'feat/p1-concurrency-bench' into feat/p1-proof 2026-09-26 01:38:17 -05:00
osobh f7c362cef5 Merge branch 'feat/p1-wasm' into feat/p1-proof 
# Conflicts:
#	Cargo.toml
#	scripts/ci-test.sh
 2026-09-26 01:38:17 -05:00
osobh 13c095a3da Merge branch 'feat/p1-h5-tools' into feat/p1-proof 
# Conflicts:
#	docs/known-issues.md
 2026-09-26 01:38:06 -05:00
osobh 591aa71d12 Merge branch 'feat/p1-plugin-filters' into feat/p1-proof 
# Conflicts:
#	crates/clawhdf5/tests/h5py_chunked_read_tests.rs
#	docs/known-issues.md
 2026-09-26 01:37:58 -05:00
osobh b9a2ce3077 Merge branch 'fix/p1-header-hardening' into feat/p1-proof 2026-09-26 01:37:43 -05:00
osobhandClaude Opus 5.5 743c32b512 docs: record the chunk dimension width libhdf5 2.0.0 refuses and we read 
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 01:32:09 -05:00
osobhandClaude Opus 5.5 993214723e test: a v2 header message running into the checksum is refused, as in libhdf5 
The review read libhdf5's H5O__chunk_deserialize as accepting a v2
message that runs up to 4 bytes into the chunk's checksum, since it
bounds message bodies by the whole chunk buffer. It does not accept it:
the message loop stops at the checksum, and the checksum read that
follows starts past it and overruns the chunk ("ran off end of input
buffer while decoding"). h5py refuses such files whether the message
runs 1, 4 or 5 bytes in, and so does clawhdf5, with its own error text.
No code change; the test pins the agreement and a comment records why.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 01:32:00 -05:00
osobhandClaude Opus 5.5 3938f7f8a2 fix(io): refuse truncated files in the VOL, async and MPI readers 
The truncated-file check and the end-of-file clamp reached File,
LazyFile and MmapFile but not clawhdf5-io's readers, which still opened
truncated files and read past the recorded end of file. NativeVol
(open, and read_dataset for from_bytes), AsyncHDF5File::from_bytes and
MpiVol's collective read now view the file through the new
vol::hdf5_view: from the superblock to Superblock::data_end, refusing a
file shorter than that.

MpiVol's read is compiled only with the mpi-io feature, which needs an
MPI installation; it was not built here. The edit there only swaps its
two-line superblock setup for hdf5_view.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 01:30:56 -05:00
osobhandClaude Opus 5.5 dd40bea467 fix(format): refuse a chunk layout whose element size is not the datatype's 
A chunked layout records the element size as its last dimension, and
libhdf5 refuses a dataset whose datatype has another size
(H5D__chunk_set_sizes: "stored datatype size in chunk layout does not
match datatype description"). clawhdf5 ignored the recorded size and
read the chunks anyway, for v3 and v4 layouts. The check runs on every
chunked read (read_chunked_data*, read_raw_data_selection) and compares
against the stored size: a variable-length element is 4 + offset size
+ 4 bytes, not Datatype::type_size's 16.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 01:29:26 -05:00
osobhandClaude Opus 5.5 afae86f3ea fix(format): write layout v4 chunk dimensions in the fewest bytes 
libhdf5 encodes a version-4 layout's chunk dimensions in (log2(max) +
8) / 8 bytes, and HDF5 2.0.0 (h5py 3.16) refuses any other width:
"stored chunk dimension encoding length does not match value calculated
from chunk dimensions". The writer rounded 3 bytes up to 4, so h5py
could not open a dataset we wrote with a chunk dimension from 65 536 to
16 777 215, for every chunk index (single chunk, fixed and extensible
array, v2 B-tree). The three encoders now share push_v4_chunk_dims,
which writes the exact width.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 01:27:22 -05:00
osobhandClaude Opus 5.5 f713847e65 fix(format): read layout v4 chunk dimensions of any width from 1 to 8 bytes 
A version-4 layout stores every chunk dimension in the fewest bytes that
hold the largest one (H5D__chunk_set_sizes: (log2(dim) + 8) / 8), so a
chunk dimension of 65 536 to 16 777 215 takes 3 bytes. Only widths 1, 2,
4 and 8 were decoded; an h5py file with chunks=(70000,) and
libver='latest' failed with UnexpectedEof. Widths 1-8 are decoded now;
0 and more than 8 are refused with libhdf5's "encoded chunk dimension
size is too large", and a dimension past u32 is refused, not truncated.

The review asked for libhdf5's check that the stored width matches the
one computed from the dimensions. HDF5 2.0.0 (h5py 3.16) refuses any
mismatch, but HDFGroup/hdf5@e124c36 ("Allow reading of files with chunk
dimensions encoded using more bytes than necessary", 2026-06-05) relaxed
it to refusing only a width too small for the dimensions, which cannot
happen once the dimensions have been decoded from that width. Follow
current libhdf5: a wider-than-needed encoding is read. clawhdf5's own
writer produces such layouts (the next commit fixes that).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 01:25:43 -05:00
osobhandClaude Opus 5.5 17fc8b1964 docs: changelog and known issues for the plugin-filter review fixes 
The short-decoding chunk (wrong data, pre-existing), the Blosc header
underflow (crash) and filter 32023 registration, each with its date and
what it changes; the conformance count is unchanged at 573 of 697.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 01:24:18 -05:00
osobhandClaude Opus 5.5 9238605661 fix(format): measure compound members by their stored size 
The compound overlap check measured each earlier member with
Datatype::type_size, which is a fixed 16 for a variable-length type. On
disk a VL member takes 4 + offset size + 4 bytes, 12 in a file with
4-byte offsets, so a member right after one was refused as "member
overlaps with previous member" (and with the type, every attribute of
the object). libhdf5 measures members by their decoded, stored size
(times a v1 member's array dimensions); so does this now.

Reading VL values in such files is a separate, older gap, now recorded
in known-issues.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 01:24:17 -05:00
osobhandClaude Opus 5.5 738b9491b2 fix(format): a codec can be registered for filter 32023 (Granular BitRound) 
With the pcodec feature, 32023 was a built-in entry (the legacy reader for
the pcodec chunks clawhdf5 <= 2.7.0 wrote under that ID), so
register_filter(32023, ...) was refused as "built in", although
UnsupportedFilter(32023) names Granular BitRound as not implemented and
the registry docs point to register_filter for such IDs.

That entry is now shared: it claims only chunks whose filter is named
"pcodec"; any other chunk with ID 32023 goes to the registered codec (or,
with none registered, gets UnsupportedFilter as before), and writing
32023 uses the registered codec. Every other built-in ID still refuses
registration.

Test: a_codec_can_be_registered_for_granular_bitround (registers, round-
trips chunks with no name and other names, still reads a legacy "pcodec"
chunk with the built-in reader, and after unregistering reads nothing).
It fails without the change ("filter 32023 ... is built in and cannot be
re-registered"). It and the existing legacy-pcodec test share a lock,
since the registry is process-wide.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 01:23:58 -05:00
osobhandClaude Opus 5.5 e10df68ed8 ci: show h5dump's version; h5rs dump output checked against 1.14.5 
dump_matches_h5dump requires byte identity with the h5dump on PATH, and
CI's rust:latest (Debian 13.7) installs hdf5-tools 1.14.5, not the 1.14.6
the test was written against. Ran the whole clawhdf5-tools suite in
rust:latest with Debian's hdf5-tools and pip h5py 3.16.0 (HDF5 2.0.0), as
CI sets it up, with CLAWHDF5_REQUIRE_INTEROP=1: 18 of 18 pass, so the
comparison needs no loosening. The version is now printed in the CI log
so a future Debian update that changes the output is easy to spot.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 01:23:52 -05:00
osobhandClaude Opus 5.5 c4d96c1390 fix(tools): h5rs dump shows NUL padding in nested strings, like h5dump 
A null-padded fixed string inside a compound or an array member printed
trimmed ("" for three NULs, "a" for "a\0b"), where h5dump prints every
byte ("\000\000\000", "a\000b"); only top-level strings were shown in
full. DATA blocks now render elements through one function that keeps
the padding at any depth.

The README now lists the remaining known differences from h5dump:
nested compounds print inline, and long double values are printed as
errors (exit 1) with the datatype as an H5T_FLOAT block.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 01:23:32 -05:00
osobhandClaude Opus 5.5 b8492bd28d fix(tools): h5rs check --data follows VL data into the global heap 
The README said check skips only "global heap collections other than
those a value read touches", but read_dataset returns the raw heap IDs,
so no collection was ever read: a file whose global heap collection
claims a 4 GiB object passed `check --data` with no problems, while
h5dump (and h5rs dump/diff) fail on it.

With --data, every variable-length element (strings and sequences, also
inside compounds, arrays and nested sequences) of every dataset and
attribute is followed into its collection. A collection that does not
parse, a missing heap object, or a sequence longer than its heap object
is a problem at the collection's address, once per object; the summary
counts the collections read.

Measured on tank, 2026-09-26: the 418 fully-read conformance ok files
still pass (scripts/h5rs-check-ok-files.sh --data, 0 flagged), and
`check --data` now flags 152 of the 180 CVE-corpus files (was 147); of
the 28 it passes, h5dump 1.14.6 rejects 21 (was 26 of 33).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 01:22:27 -05:00
osobhandClaude Opus 5.5 a5bd70216c fix(format): a chunk that decodes short is an error, not zero-filled 
HDF5 stores every chunk at the full chunk size (edge chunks are padded
before filtering, and with "don't filter partial edge chunks" they are
stored raw at full size), so a filter pipeline that decodes to fewer bytes
means a corrupt chunk. Every chunk reader padded it with zeros and
returned it as data. libhdf5 returns the rest uninitialised, or fails when
the filter checks (Blosc with nbytes = 0).

New filters::decompress_chunk_exact decodes and then requires exactly the
chunk size, with the chunk's coordinates in the error
(ChunkedReadError "chunk at [16] decoded to 16 bytes, expected 32"). It
replaces decompress_chunk_masked at every chunk read path: the full read
(sequential and lane-partitioned), the cached read, the sweep read, the
planned-selection read, parallel_read's three decoders and partial_read's
box read. decompress_chunk_masked is unchanged (fractal-heap huge objects
already checked their own size). Blosc also rejects a frame declaring no
data where the chunk size is known.

Tests, each failing with the check disabled: filters and parallel_read
unit tests; h5py_short_decoded_chunk_is_an_error (gzip chunks rewritten
short with write_direct_chunk: 1-D, a 2-D edge chunk, and 40 chunks with
shuffle, read through File full/cached/selection reads, a selection that
avoids the chunk still reads, MmapFile and LazyFile, with and without the
parallel feature); plugin_filters_interop short_decoding_chunks_are_errors
(Blosc nbytes=0 and short, LZF and bzip2 short; the Blosc nbytes=0 case
read as 16 zeros before). The existing don't-filter-partial-edge-chunks
tests still pass. Conformance (tank, 2026-09-26): 573 of 697 ok, and no
file changed class, reader result or first issue against the pre-fix run.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 01:22:26 -05:00
osobhandClaude Opus 5.5 17f09375ad fix(format): refuse to write datatypes the reader refuses 
The reader now refuses a compound with a repeated field name or no
fields and an enum member with an empty name, as libhdf5 does, but the
writer still wrote them: CompoundTypeBuilder and EnumTypeBuilder build
them without complaint, so clawhdf5 wrote files it could not read back.
They were never valid HDF5; h5py refuses them.

Datatype::check_encodable, which FileWriter::finish runs on every
dataset and attribute type, now parses the type's own encoding back and
refuses one the reader refuses, with the reader's reason. That keeps the
writer in step with every reader check, not only these three.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 01:21:20 -05:00
osobhandClaude Opus 5.5 386bd1d41e fix(tools): h5rs diff names its options as h5diff does 
-c meant "list at most N differences" in h5rs, but in h5diff -c is
--compare (a flag) and the count is -n/--count=N, so a script moved over
from h5diff behaved differently: `h5diff -r -c 2 A B` exits 2 (the 2 is
taken as a file name) while h5rs exited 1.

The count is now -n/--count, -c/--compare is accepted (h5rs always lists
objects that are not comparable), and the --count=N, --delta=D,
--relative=R forms are accepted; exit codes equal h5diff's on 7 cases.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 01:18:56 -05:00
osobhandClaude Opus 5.5 b5e43bacd7 fix(tools): h5rs diff compares soft links by target, like h5diff 
An OBJ that was a soft link was resolved and its target object compared,
so two files whose /g/s both point at /z differed when /z did: exit 1,
where h5diff (without --follow-symlinks) compares the links' target paths
and exits 0.

A soft link is now compared as a link wherever it is, OBJ included.
--follow-symlinks compares the objects soft links lead to instead, walks
into soft-linked groups, resolves relative targets against the link's
group, and treats two dangling links as the same; exit codes equal
h5diff's on 14 cases. External links are never followed (documented).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 01:18:13 -05:00
osobhandClaude Opus 5.5 a14ccc36bf fix(format): limit chunks to 4 GiB only under a v1 B-tree index 
libhdf5 refuses a chunk of 4 GiB or more only when a version-1 B-tree
indexes it (H5D__chunk_init: "chunk size must be < 4GB with v1 b-tree
index"). HDF5 2.0 writes larger chunks with layout version 5, and h5py
reads them; these were refused. chunk_geometry now takes the layout
version and applies the limit to layout version 3 and earlier only.

The interop test is ignored by default: h5py writes a 4 GiB chunk and
both libraries hold it in memory.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 01:17:36 -05:00
osobhandClaude Opus 5.5 7f52a6f3ba test(format): fuzz every plugin-filter decoder for panics 
Audited LZF, bitshuffle, bzip2 and Blosc/BloscLZ for arithmetic on
header fields and unchecked slicing. The only live bug was the Blosc
frame-size underflow fixed in the previous commit; bzip2's output-growth
step now uses a saturating subtraction as well (the allocator may hand
back more capacity than asked for).

src/test_fuzz.rs (tests only) feeds each decoder random bytes, truncated
seeds and one-to-four-edit mutations of valid frames, biased towards
edge-case u32 values in size and offset fields, and asserts no panic and
no output over the limit (tests build with overflow checks and debug
assertions). Per decoder: LZF, bzip2, bitshuffle in all six mode/block
settings plus hostile cd_values, Blosc across four codecs, three shuffles,
stored frames and a hand-built BloscLZ frame, and BloscLZ streams alone.
With the previous commit's check removed, fuzzed_frames_never_panic panics
at the same subtraction. A 100x-iteration soak (different seed) found no
other panic.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 01:16:44 -05:00
osobhandClaude Opus 5.5 f325d111f3 fix(tools): h5rs diff compares integers exactly under -d/-p 
With a tolerance, integers were converted to f64 before comparing, so
int64/uint64 values above 2^53 that differ compared equal: -d 0 on 2^60
and 2^60 + 1 exited 0, where h5diff exits 1. Integer pairs are now
compared in i128 (the delta against floor(D), the relative quotient from
an exact difference), and the report prints the exact difference.

h5diff compares exactly when -p is below the f64 epsilon (2^60 and
2^60 + 1 differ at -p 1e-18, and nextafter(2, 0) and 2 at -p 1.5e-16);
h5rs now does the same.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 01:16:09 -05:00
osobhandClaude Opus 5.5 699ee9c447 fix(tools): h5rs diff compares every name of a hard-linked object 
The path walk skipped the second hard link to an object, so a file that
shares one dataset between /x and /y differed from a file holding two
identical copies: "</y> exists only in <B>", exit 1, where h5diff exits 0.
For a hard-linked group every member was reported the same way.

diff now enumerates every path below the start object (a hard link back
to an ancestor is recorded but not descended into), so each name is
compared. A group whose links cannot be read is now an error instead of
an empty group.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 01:14:53 -05:00
osobhandClaude Opus 5.5 9416c58723 fix(format): a Blosc frame shorter than its header is an error, not a panic 
A hostile chunk whose header gave a compressed size below 16 bytes, not
stored raw, made the block-table check subtract past zero: a panic in any
build with overflow checks (cargo test, maturin develop, debug CLI). The
frame size is now checked against the header size, and the stream-length
read no longer adds to an untrusted offset.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 01:14:33 -05:00
osobhandClaude Opus 5.5 6a8ee3ec7f docs: changelog and known issues for the header hardening 
CHANGELOG (Correctness): the new header, datatype, chunk and truncation
checks, what is left out on purpose (checks HDF5 2.0 lacks; the two
v2.7.0 writer quirks), the conformance numbers and the new FormatError
variants. known-issues: the "Header checks" audit gap is fixed, with the
one CVE object and two CVE files libhdf5 still refuses and we read.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 00:36:49 -05:00
osobhandClaude Opus 5.5 a59d83d47d test: compare header and datatype damage with what h5py refuses 
h5py writes a dataset (libver earliest, so version-1 object headers) and
the script damages one field of a copy: a layout message flagged
shareable, a message size that is not a multiple of 8, a compound field
that repeats an earlier name or overlaps it, an empty enum member name, a
float exponent overlapping the mantissa. h5py refuses every damaged copy,
and clawhdf5 must refuse exactly those and read the valid files. All of
them but the unaligned one (then an UnexpectedEof) read before this
branch. The helper now reads any datatype
(File::read_multi) rather than only integers.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 00:35:41 -05:00
osobhandClaude Opus 5.5 bb39be7f24 fix(format): keep reading the floats and empty strings clawhdf5 v2.7.0 wrote 
Two of the datatype checks added on this branch refused files clawhdf5
itself wrote up to v2.7.0: it put the sign bit of every float at
position 63 (so every f32 it wrote failed "sign bit position out of
bounds", including every agent store's embeddings), and wrote an
empty-string attribute with a size-0 string type ("invalid datatype
size", failing every attribute of the object). libhdf5 refuses both, but
neither decodes to wrong values (an IEEE float's sign position is not
used; a size-0 string is empty), so this reader keeps accepting them.

New fixtures written by clawhdf5 v2.7.0 (FileBuilder with every datatype,
layout and attribute kind it could write, and a FileWriter paged file)
and legacy_writer_files.rs, which reads every object of them. The agent's
v2.5.0 store fixture (float16_store) passes again.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 00:34:44 -05:00
osobhandClaude Opus 5.5 845a9d0125 docs: h5rs check inherits the library's header-check gap 
Measured on the CVE corpus: check --data passes 33 of 180 files, and
h5dump 1.14.6 rejects 26 of those. Recorded under the open "Header checks"
gap and in the crate README.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 00:33:06 -05:00
osobhandClaude Opus 5.5 7d7a7e75d4 fix: refuse truncated files and read nothing past the recorded end of file 
The superblock records where the file's data ends. libhdf5 refuses to
open a file shorter than that ("truncated file", H5F__super_read) and
fails any read past it ("addr overflow" / "address plus size exceeds
file eoa"). clawhdf5 read whatever was left of a truncated file, and read
bytes after the recorded end as if they belonged to the file.

New Superblock::data_end: FormatError::TruncatedFile for a file shorter
than its recorded end, otherwise where the HDF5 data ends. As in libhdf5,
the recorded end moves with the superblock when its recorded base address
is not where it is (a user block added afterwards; cve-2021-36977, which
h5py reads, depends on it), and the check is skipped for a v3 superblock
still being written in SWMR mode. File, LazyFile, MmapFile and the
conformance probe refuse a truncated file and parse only up to the end.

Files clawhdf5 writes record their true length, and the v2.5.0 agent-store
fixture and files written by v2.7.0 (plain, paged, user-block free) pass
the check.

Interop test: h5py writes a file; a copy missing its last 8 bytes must be
refused by both, a copy with bytes appended and one moved behind a new
512-byte user block must read in both.

Conformance (cached corpus, tank): 570 -> 571 ok
(h5clear_fsm_persist_less.h5, whose data past the recorded end was being
read); ten files h5py refuses as truncated (cve-2018-13874,
cve-2018-13876, the family/multi/subfiling members, h5clear_fsm_persist_
greater/user_greater) are now refused at open instead of read.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 00:30:40 -05:00
osobhandClaude Opus 5.5 0685037593 docs: h5rs in the changelog and the crate table 
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 00:29:38 -05:00
osobhandClaude Opus 5.5 e92faa23a6 ci: install hdf5-tools for the h5rs interop tests 
The clawhdf5-tools interop tests compare h5rs with h5ls, h5stat, h5dump and
h5diff, and CLAWHDF5_REQUIRE_INTEROP=1 turns a missing tool into a failure.
Also hold clawhdf5-tools to the no-C-in-the-default-build check.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 00:29:38 -05:00
osobhandClaude Opus 5.5 40968b3578 test(tools): h5rs sweeps over the conformance corpora 
scripts/h5rs-fuzz.sh runs every h5rs subcommand over every file of a corpus
(default: the HDF Group's CVE reproducers), optionally with byte-flipped
copies (MUTATE=N), under a timeout and a memory limit, with a debug build so
integer overflow panics instead of wrapping; any exit status above 2 (a
caught panic, a timeout, a signal) fails it. It found size*8 overflows in
the datatype names on cve-2021-46244.h5, cve-2024-29161.h5 and unknown-1.h5
(fixed in the crate before it landed).

scripts/h5rs-check-ok-files.sh runs check (--data) over the conformance
files that clawhdf5 and h5py both read in full; none may be flagged.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 00:29:34 -05:00
osobhandClaude Opus 5.5 310448bfcb feat(tools): h5rs, pure-Rust HDF5 tools (ls, dump, stat, diff, check) 
New workspace crate clawhdf5-tools with one binary, h5rs, built only on the
clawhdf5 facade and clawhdf5-format (no libhdf5, no C):

- ls [-r] [-v] FILE[/path]: h5ls's listing (same text in its first two
  columns) plus the datatype; -v adds address, link count, layout and chunk
  index, chunk size, storage, filters, datatype and attributes.
- dump [--json] [-A] [-p] [-d PATH] FILE: h5dump DDL (byte-identical to
  h5dump 1.14.6 on the test files) or hdf5-json.
- stat FILE: h5stat's object/link/rank/layout/filter/attribute counts, raw
  data and total size.
- diff [-r] [-q] [-d D] [-p R] A B [OBJ1 [OBJ2]]: structural and value
  differences, exit 0/1/2 like h5diff.
- check [--data] FILE: walks every object, parses every message, verifies
  the checksums of every v2+ structure (including the fractal heap blocks
  the library never checks), checks chunk indexes against their datasets
  and raw data for out-of-file or overlapping extents; every problem with
  its address.

Values over --max-bytes are reported, not read; dense-storage heaps are
verified before objects are read from them; panics are caught (exit 3).
Tests compare with h5ls, h5stat, h5dump and h5diff and with h5py's values,
and flip the checksum of every checksummed structure in a v1.14-format file.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 00:29:28 -05:00
osobhandClaude Opus 5.5 e73ac2af09 fix(format): validate chunk dimensions and chunk index offsets like libhdf5 
A chunk dimension of 0 read a dataset as all fill values, 0x80000000 made
an 8 GiB chunk, and a chunk dimension the chunk index's offsets are not
multiples of read chunks at the wrong place (cve-2018-11205). libhdf5
refuses all of these; now so does clawhdf5:

- DataLayout::parse (H5O__layout_decode): no chunk dimension 0 ("bad chunk
  dimension value"), at most 33 dimensions, and before layout v4 at least
  2 ("bad dimensions for chunked storage"). New
  FormatError::InvalidChunkDimensions.
- Reading a chunked dataset (H5D__chunk_init / H5D__chunk_set_sizes): the
  chunk rank must match the dataspace's and a chunk must be under 4 GiB.
  One chunked_read::chunk_geometry replaces the four copies of the rank
  check.
- v1 B-tree chunk index (H5D__btree_decode_key): every key's offsets must
  be multiples of the chunk dimensions, including the keys that only bound
  a node, which is where cve-2018-11205's bad dimension shows. New
  chunked_read::collect_chunk_info_checked; the chunked read and selection
  paths use it.

New interop test header_validation_interop.rs: h5py writes chunked files
(layout v3 and v4), the script corrupts the chunk dimension, and
clawhdf5 must read exactly the copies h5py reads.

Conformance (cached corpus, tank): 570 ok, unchanged; cve-2018-11205 now
refuses the dataset h5py refuses; six more objects that already failed now
fail with libhdf5's reason.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 00:21:58 -05:00
osobhandClaude Opus 5.5 e01160299a docs: plugin filters and the filter registry 
CHANGELOG (Unreleased): LZF, bitshuffle, bzip2 and Blosc read and write
in pure Rust, their features, the ChunkOptions::plugin field (breaking for
struct-literal construction), the filter registry, the named
UnsupportedFilter message, and Blosc2/ZFP still unimplemented.
README: the clawhdf5-format feature table gains lzf (default),
bitshuffle, bzip2, blosc and plugin-filters, with how to write them and
what is not implemented; no speed claims. docs/known-issues.md: the audit's
filter gap is marked fixed 2026-09-26 for LZF/bitshuffle/bzip2/Blosc,
Blosc2 and ZFP still open. CLAUDE.md: the clawhdf5-filters row no longer
says "No Blosc". clawhdf5-format's crate docs list the new features.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 00:14:47 -05:00
osobhandClaude Opus 5.5 5461a13984 ci: js-sys is not C in the no-C check 
The check matches any *-sys crate, and clawhdf5-wasm pulls in js-sys,
wasm-bindgen's bindings to JavaScript, which compiles no C. Exempt it
by name so the check keeps catching real C for the wasm crate.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 00:14:30 -05:00
osobhandClaude Opus 5.5 056092b082 ci: lint and test the plugin filters; the conformance probe reads them 
scripts/ci-test.sh: the format feature matrix (clippy and tests) adds
plugin-filters; bitshuffle, bzip2 and blosc are each linted alone (blosc
and bitshuffle share code); the facade is linted with plugin-filters; and
the interop section runs tests/plugin_filters_interop.rs with it, against
h5py + hdf5plugin (CI's venv already installs hdf5plugin).

conformance/probe enables plugin-filters. Sweep (tank, 2026-09-26,
conformance/run.sh --no-fetch against the cached corpus): 573 of 697 ok
(baseline 569), no regressions; newly ok: h5ex_d_blosc.h5,
h5ex_d_bshuf.h5, h5ex_d_bzip2.h5, h5ex_d_lzf.h5. h5ex_d_blosc2.h5 and
h5ex_d_zfp.h5 remain UnsupportedFilter.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 00:14:28 -05:00
osobhandClaude Opus 5.5 a5ca970015 fix: refuse numeric types with unusually many unused bits in v1 headers 
libhdf5 1.14.4+ treats an integer, float or bit field wider than a byte
whose precision and offset leave more than half its bits unused as
corruption when the type sits in a header without a checksum (version 1),
unless the file is opened with H5Pset_relax_file_integrity_checks
(H5T_is_numeric_with_unusual_unused_bits). clawhdf5 read such types,
e.g. a 3-bit integer in 4 bytes (cve-2024-29162) or a 32-bit float in
65525 bytes (cve-2024-32614, tmisc38a.h5).

New Datatype::check_unused_bits (recursive) and Datatype::parse_in_header,
which applies it for version-1 headers. Dataset datatypes (facade File,
LazyFile, MmapFile; clawhdf5-io VOL, MPI VOL, async reader; the
conformance probe) and compact attributes in version-1 headers use it.

Conformance (cached corpus, tank): 570 ok, unchanged; cve-2024-29162,
cve-2024-32614 and tmisc38a.h5 now refuse the object h5py refuses, and
tmisc38b.h5 / unknown-1.h5 now fail with libhdf5's reason.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 00:12:04 -05:00
osobhandClaude Opus 5.5 e7a7951f1e feat(format): name the filter in UnsupportedFilter errors; Blosc2/ZFP stay errors 
Blosc2 (32026) is out of reach for now: hdf5plugin's Blosc2 filter stores
each HDF5 chunk as a Blosc2 super-chunk frame (msgpack header, a compressed
chunk-offset index, trailer metalayers) and, for 2-D and larger chunks, as
a B2ND array whose n-D blocks have to be reassembled - on top of the Blosc2
chunk format itself (extended header, filter pipeline, special-value
chunks). ZFP (32013) is out of scope. Both keep failing with
UnsupportedFilter, and the message now says what the ID is:
"unsupported filter: 32026 (Blosc2, not implemented by clawhdf5)", or,
for a filter this build left out, "... (Blosc; this build lacks the
`blosc` feature)". filter_registry::known_filter exposes the table.

tests/plugin_filters_interop.rs: hdf5plugin's Blosc2 and ZFP datasets
read as an error naming the filter, never as data.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 00:11:57 -05:00
osobhandClaude Opus 5.5 1f71f3bcbc feat(format): Blosc filter (32001), read and write, pure Rust 
hdf5plugin's Blosc (hdf5-blosc) failed with UnsupportedFilter(32001). The
new `blosc` feature decodes the Blosc 1 frame c-blosc 1.x writes: the
16-byte header, raw ("memcpyed") frames, the block table, blocks split
into one stream per byte plane (and the "do not split" flag), streams
stored raw, the byte shuffle and bit shuffle (whole 8-element groups, the
rest copied) - and every codec hdf5plugin offers: BloscLZ (implemented
here from c-blosc 1.21's blosclz_decompress, including its rejection of
malformed and truncated streams), LZ4/LZ4HC (lz4_flex), Snappy (snap),
Zlib (flate2) and Zstandard (ruzstd). Every stream must decode to exactly
its size and the frame to at most the chunk size; a frame of another
format version (Blosc 2) is a clear error.

It also encodes (DatasetBuilder::with_blosc(codec, level, shuffle)):
LZ4, Snappy, Zlib or Zstandard, with c-blosc's split rule, raw streams
where compression does not pay, and a stored frame for level 0 or
incompressible data. It cannot write BloscLZ (asking for it is an
error). `plugin-filters` enables LZF, bitshuffle, bzip2 and Blosc.

Interop: hdf5plugin writes all six codecs x {no, byte, bit} shuffle at
levels 5/9/1, plus level 0, over the 12-case matrix, read byte for byte;
our four codecs x four shuffle/level settings read back through
hdf5plugin. Both fail with the decoder removed. `cargo tree` with
`plugin-filters` has no -sys crate other than libbz2-rs-sys (pure Rust),
no cc and no cmake.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 00:10:05 -05:00
osobhandClaude Opus 5.5 3cf8cd86f2 fix(format): refuse datatypes libhdf5 refuses to decode 
Datatype::parse now makes the checks of libhdf5's H5O__dtype_decode_helper
and fails with InvalidDatatype (libhdf5's own error text) instead of
decoding a corrupt type:

- size 0 ("invalid datatype size"), for every class;
- integer bit offset/precision outside the type, or precision 0;
- float sign/exponent/mantissa outside the type, empty, or overlapping;
  normalization 3; bit 6 without bit 0 from version 3;
- compound with no members, a member outside the compound, a duplicate
  name, or a member overlapping an earlier one;
- enum whose size differs from its base type's, or an empty member name;
- array of more than 32 dimensions or with a zero-sized one (v1 compound
  array members now say so rather than InvalidDatatypeVersion);
- opaque tag length that is not a multiple of 8.

Bit 6 of a version-1/2 float's class bits used to be read as VAX order,
byte-swapping values; libhdf5 ignores it before version 3, and so does
this now.

Only checks HDF5 2.0 (h5py 3.16) makes are added: newer libhdf5 also
checks bit fields, the variable-length kind and array sizes, but h5py
opens files that fail those, so they are left out. Each check was
confirmed against h5py by corrupting a file it wrote.

The conformance probe now decodes committed datatypes, as h5py's f[name]
does. Conformance (cached corpus, tank): 570 ok, unchanged. Objects
libhdf5 refuses that clawhdf5 used to read: cve-2016-4332-mtime (/cmpnd),
cve-2017-17508, cve-2024-32616 (/type1), cve-2024-32618, cve-2026-34734,
bad_compound.h5 (/cmpnd, /dataset); eight more that already failed now
fail with libhdf5's reason (e.g. cve-2024-29163 "mantissa range out of
bounds").

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 00:08:58 -05:00
osobhandClaude Opus 5.5 34987ec194 docs: clawhdf5-wasm in the changelog, known issues and CLAUDE.md 
Changelog entry with the sizes measured on tank on 2026-09-26 (and the
h5wasm 0.10.3 comparison), the browser build's limits as a known-issues
entry, and where its tests run.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 00:06:58 -05:00
osobhandClaude Opus 5.5 b58d61cfb7 test(wasm): accept a zstd read when the build has the filter 
cargo test --workspace unifies clawhdf5-format/zstd on (another member
enables it), so the native interop test read the Zstd dataset that the
wasm build refuses. The fixture now records its values plus the error
the wasm build must give; the native test accepts either, the Node test
of the real wasm package still requires the error.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 00:06:26 -05:00
osobhandClaude Opus 5.5 1abd93e0f8 feat(wasm): examples/wasm-viewer, an HDF5/NetCDF-4 viewer page 
Drop a file (or pass ?file=<url>&path=<object>), browse the tree lazily,
see a dataset's type, shape, max shape and attributes, and page through
its values as 50x12 hyperslab windows (leading dims of 3-D+ data held
at chosen indices). build.sh produces pkg/ (not committed) with
wasm-bindgen --target web and checks the CLI matches the crate version.

test/run.sh builds it and runs test.mjs under Node against the h5py/
netCDF4 fixture (250 checks: every dataset whole and as a strided
hyperslab, listings, attributes, error paths, the page's DOM-free
helpers), then browser.sh renders the page in headless Chromium for
eight objects and checks the DOM. The fixture gains LZ4 (read) and Zstd
(refused: links C) datasets and a compound attribute (value null plus
its type). ci-test.sh runs it when node and wasm-bindgen exist; the CI
container has neither, so CI relies on the native h5py_interop test.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 00:05:34 -05:00
osobhandClaude Opus 5.5 6dfd239011 feat(format): bzip2 filter (307), read and write, pure Rust 
hdf5plugin's BZip2 failed with UnsupportedFilter(307). The new `bzip2`
feature decodes the single bzip2 stream H5Zbzip2.c stores, bounded by the
chunk size (a truncated stream is an error, not short data), and encodes
at block size cd_values[0] (DatasetBuilder::with_bzip2(level)). It uses the
bzip2 crate's default backend, libbz2-rs-sys, a pure-Rust port of
libbzip2: `cargo tree` shows no cc/cmake, and nothing is compiled from C.

Interop: hdf5plugin writes block sizes 9, 1 and 5+shuffle over the
12-case matrix, read byte for byte; ours at 9 (shuffled) and 1 (not)
reads back through hdf5plugin. Both fail with the decoder removed.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 00:05:26 -05:00
osobhandClaude Opus 5.5 07094e34a9 feat(format): bitshuffle filter (32008), with its LZ4 and Zstandard modes 
hdf5plugin's Bitshuffle failed with UnsupportedFilter(32008). The new
`bitshuffle` feature (pure Rust: lz4_flex, and ruzstd for Zstandard — the
`zstd` feature's libzstd is not needed) decodes all three modes of
bshuf_h5filter.c — transpose only, LZ4 and Zstandard blocks behind the
12-byte header — including the default and explicit block sizes, the
shorter last block rounded down to a multiple of 8 elements, and the
untransposed trailing elements. Sizes read from the chunk are bounded by
the chunk size.

It also encodes: DatasetBuilder::with_bitshuffle(BitshuffleCompression)
or PluginFilter::Bitshuffle { block_size, compression } writes the filter
with hdf5plugin's cd_values and no automatic byte shuffle. ruzstd has one
compression level (about zstd's 1); the requested level is recorded.

The bit transpose is checked bit for bit against a one-bit-at-a-time model
(which matched hdf5plugin's output) and is shared with blosc next.
Interop: hdf5plugin writes none/LZ4/Zstandard at default and explicit
block sizes and levels over the 12-case matrix, read byte for byte; our
three modes at two block sizes read back through hdf5plugin. Both fail
with the decoder removed.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 00:04:05 -05:00
osobhandClaude Opus 5.5 f4dee1cd08 fix(format): refuse object headers libhdf5 refuses to load 
ObjectHeader::parse now checks each header message the way libhdf5's
H5O__chunk_deserialize does, and fails with InvalidObjectHeader (libhdf5's
own error text) instead of reading objects out of a corrupt header:

- v1: every message in chunk 0 is read (not just the prefix's count) and
  more messages than the prefix claims is "bad object header message
  count"; message sizes must be multiples of 8; leftover bytes are a gap,
  which only v2 allows; the prefix's chunk size must fit its count.
- v1 and v2: a message running past its chunk is an error (it used to end
  the chunk quietly, dropping it and everything after); contradictory
  message flags; a message of a class that cannot be shared flagged
  shared/shareable; a reference-count message in a v1 header; malformed
  continuation, reference-count and modification-time messages (libhdf5
  decodes these while loading the header).
- v2: unknown header status flags, max_compact < min_dense, a chunk 0
  smaller than a message header, a gap in a chunk that has NIL messages.

Conformance (cached corpus, tank): 569 -> 570 ok (h5stat_err_refcount.h5).
Objects libhdf5 refuses that clawhdf5 used to read: cve-2016-4332-mtime
(/dataset), cve-2016-4332-mtime-new, cve-2018-11204, cve-2018-13873,
cve-2024-32619, cve-2024-33873, cve-2024-33874, gh-4433-poc-08; seven more
CVE objects that already failed now fail with libhdf5's reason.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 00:02:49 -05:00
osobhandClaude Opus 5.5 e38f9123db feat(format): LZF filter (32000), read and write, pure Rust 
h5py's built-in compression="lzf" failed with UnsupportedFilter(32000). The
new `lzf` feature (no dependencies, on by default in clawhdf5-format and
the facade) decodes the raw liblzf stream h5py's filter stores, bounded by
the chunk size, and encodes it: DatasetBuilder::with_lzf() (or
with_plugin_filter(PluginFilter::Lzf)) writes the filter with h5py's
cd_values (filter version 4, liblzf 0x0105, chunk size in bytes), flagged
optional as h5py does. ChunkOptions gains a `plugin` field for the plugin
filters; build_pipeline_for_chunk passes the chunk size to filters that
record it.

tests/plugin_filters_interop.rs: h5py writes LZF (alone, with shuffle, with
shuffle+fletcher32) over 12 dtype/shape/chunk/data cases with partial edge
chunks and incompressible data, and every dataset reads byte for byte equal
to its unfiltered twin; our LZF output (1-D and 2-D, edge chunks, with and
without shuffle) reads back in h5py. Both fail with the decoder removed.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 00:01:22 -05:00
osobhandClaude Opus 5.5 a42b646689 feat(wasm): clawhdf5-wasm, the reader for JavaScript via wasm-bindgen 
open(bytes) -> H5File with kind/list/info/attrs/attrErrors/read/
readHyperslab. Numeric data comes back in the typed array of the
stored width (Int16Array for i16, BigInt64Array for i64, Float32Array
for f32/f16, ...), strings and enum names as string arrays, array
datatypes flattened with their dims appended to the shape. Compound,
reference, opaque and VL-sequence datasets are refused with an error
naming the type; nothing is returned as reinterpreted bytes.

The logic is in a plain-Rust core module, tested natively: unit tests,
and h5py_interop, which compares every dataset, hyperslab, listing and
attribute of an h5py- and a netCDF4-written file with what libhdf5
reads back (generator shared with the Node test of the built package).

No mmap, no threads; lz4 is on, zstd/szip (C) are not. A
wasm-release profile (opt-level s, LTO) serves the browser build.
ci-test.sh lints the crate for wasm32 and checks it builds no C.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 00:00:46 -05:00
osobhandClaude Opus 5.5 74f9f50086 ci: build the clawhdf5 read path for wasm32-unknown-unknown 
The facade already builds for the browser target without mmap (and
with it: memmap2 compiles there and File::open just fails, as std::fs
does). Nothing needed gating; keep it that way with a ci-test.sh step,
and install the target in the CI container.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-26 00:00:11 -05:00
osobhandClaude Opus 5.5 d16544b928 feat(format): a filter registry — filters are looked up by ID 
decompress_chunk_masked and compress_chunk matched on the filter ID. They
now look the ID up in filter_registry: a static table of the built-in
filters compiled into this build (a filter whose cargo feature is off is
simply absent), then the codecs an application registered at run time with
register_filter (a FilterCodec, or a plain decoding closure). Registered
codecs cannot shadow a built-in one, and their output is held to the same
per-stage bound as the built-in decoders. An ID in neither tier still fails
with UnsupportedFilter(id).

The "feature off" stub functions that returned UnsupportedFilter are gone:
the table leaves those filters out instead.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-25 23:56:37 -05:00
osobhandClaude Opus 5.5 3b24e6753b bench: concurrent-read harness against h5py threads and processes 
concurrent_read reads one shared File from 1-16 threads: every dataset
in full (distinct datasets per thread) and random hyperslabs of one
dataset, over a deflate and a contiguous file it generates (or reuses
while manifest.json matches). It reports decoded MB/s and scaling
efficiency, warm or --cold (posix_fadvise) page cache, sizes the decode
pool with --decode-threads, and writes JSON.

scripts/concurrent_read_h5py.py runs the same workload on the same files
with h5py threads or spawned processes (same splitmix64 data and slab
stream, checked at spot elements), and compare_concurrent_read.py prints
one table and refuses runs with different workloads. A smoke test runs
all three end to end on tiny files (h5py half honours CLAWHDF5_PYTHON /
CLAWHDF5_REQUIRE_INTEROP).

BENCHMARKS.md gets a "Concurrent reads" section with the commands, marked
not yet measured.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-25 23:56:36 -05:00
osobhandClaude Opus 5.5 e815eb922f feat(facade): Dataset::raw_datatype returns the full stored datatype 
dtype() simplifies the type (no byte order, string padding or member
offsets), so callers could not decode read_selection's bytes for types
the typed read_* methods skip. raw_datatype() returns the parsed
Datatype, committed types resolved, for use with data_read.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-25 23:51:20 -05:00
osobh c9c5337a62 Merge pull request 'Reproducible HDF5 conformance sweep and nightly CI job' (#12) from feat/p1-conformance-report into main
CI / test-arm64 (push) Successful in 1m20s
Details
CI / test (push) Successful in 5m30s
Details 
Reviewed-on: #12
 2026-09-26 04:45:33 +00:00
osobhandClaude Opus 5.5 bb78d70b99 docs: changelog for the review follow-up fixes
CI / test-arm64 (pull_request) Successful in 1m7s
Details
CI / test (pull_request) Successful in 5m31s
Details 
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-25 22:46:44 -05:00
osobhandClaude Opus 5.5 a7de15534c docs: conformance report after the read-gap fixes (569 of 697 ok) 
Regenerated on tank at 10d1029: ok 467 -> 569, our-error 123 -> 14,
mismatch 15 -> 22 (six user-defined-link files moved from our-error to a
listing difference), no panics, hangs, crashes or OOM. Baseline raised.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-25 22:46:37 -05:00
osobhandClaude Opus 5.5 10d1029ead conformance: probe files with a user block and VDS the library's way 
Superblock::parse now refuses a user-block offset, and the raw read path
no longer guesses a VDS fill value. The probe looks at the file from the
superblock on and reads virtual datasets with vds::read_virtual_dataset,
the dataset's fill value and its source-derived extent.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-25 22:46:08 -05:00
osobhandClaude Opus 5.5 883980f2bd test: compare h5py's v1 compound field names with what clawhdf5 reads 
The test compared h5py against its own expected table, so it passed with
the fix reverted. Found by the adversarial review.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-25 22:44:05 -05:00
osobhandClaude Opus 5.5 d6e426e6d5 fix(agent): fail to open a store whose /meta has an unreadable attribute 
Group::attrs now leaves out an attribute it cannot decode. The agent read
its settings through it, so a store whose float16 (or compression,
quantized_index, WAL mark, signature...) attribute could not be decoded
opened with the default in its place, and no error. /meta is now read
with attrs_with_errors and any unreadable attribute is a Schema error,
as it was before attrs became tolerant. Found by the adversarial review.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-25 22:43:53 -05:00
osobhandClaude Opus 5.5 256e7b89e4 fix(format): refuse variable-length and reference VDS data from another file 
Their elements are global-heap IDs and object addresses in the source
file. The VDS reader copied them raw, so anything decoding them against
the virtual dataset's file got another object's data with no error.
Same-file sources are unaffected. Found by the adversarial review.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-25 22:42:57 -05:00
osobh f2e704abf3 Merge branch 'fix/p1-vds' into fix/p1-read-gaps 
# Conflicts:
#	CHANGELOG.md
#	crates/clawhdf5-format/src/data_read.rs
 2026-09-25 22:42:04 -05:00
osobh 61f36516d7 Merge branch 'fix/p1-attrs-links' into fix/p1-read-gaps 
# Conflicts:
#	crates/clawhdf5-format/src/attribute.rs
#	crates/clawhdf5-format/src/group_v1.rs
#	crates/clawhdf5/src/lazy.rs
#	crates/clawhdf5/src/mmap_file.rs
#	docs/known-issues.md
 2026-09-25 22:41:33 -05:00
osobh 45720fe5a6 Merge branch 'fix/p1-userblock-shared' into fix/p1-read-gaps 
# Conflicts:
#	crates/clawhdf5-format/src/attribute.rs
#	crates/clawhdf5-format/src/datatype.rs
#	crates/clawhdf5-format/src/shared_message.rs
#	docs/known-issues.md
 2026-09-25 22:40:55 -05:00
osobh adf961c883 Merge branch 'feat/p1-layout-v1v2' into fix/p1-read-gaps 2026-09-25 22:40:26 -05:00
osobhandClaude Opus 5.5 b4a44a2e66 feat(format): read unlimited and printf-style VDS mappings like libhdf5 
Unlimited VDS mappings were refused, and printf-style source names
("f-%b.h5") were not expanded, so those regions read as fill (read-matrix
case 0470: 29 of 30 values wrong). All 7 virtual datasets in the libhdf5
test set use such mappings.

Implement H5Dvirtual.c's semantics in the vds module:
- %b is the block number, %% a literal %, other specifiers are an error;
  block j of the virtual selection comes from the source named with j,
  probing from 0 to the first missing source (printf gap 0);
- unlimited source/virtual selections are clipped to what the source's
  current extent fills (H5S_hyper_get_clip_extent_match, partial last
  block included);
- the extent is recomputed as H5Dget_space does (view "last available":
  the largest clip, never below what limited mappings need), exposed as
  vds::virtual_dataset_extent and used by Dataset::shape();
- a source in the other byte order is byte-swapped; other conversions stay
  an error.

Tests: vds_interop::vds_printf_source_names,
vds_unlimited_mappings_follow_source_extents (h5py low-level API, earliest
and latest format) and vds_libhdf5_test_files (vds-eiger, 4_vds and
vds-percival-unlim-maxmin from HDF5's tools/test/testfiles/vds, committed
as fixtures) all compare shape and values with h5py; unit tests for the
clip arithmetic, name parsing and mapping rules.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-25 22:10:09 -05:00
osobhandClaude Opus 5.5 17fa783dce fix(format): resolve SOHM-shared messages on every path 
A message shared through the file's SOHM heap (H5Pset_shared_mesg_index)
is referenced by heap ID, which needs the SOHM table from the superblock
extension. Only message_data_with_sohm (used for fill values) loaded it;
resolve_shared_message passed no table, so a SOHM-shared datatype,
dataspace, filter pipeline or attribute failed with "invalid shared
message version: 2" and the dataset or attribute could not be read.
resolve_shared_message now loads the table when the reference carries a
heap ID.

Found while making attrs() tolerant: SOHM attributes turned from an
error into missing keys in the audit read matrix. With this fix all 36
SOHM cases there match h5py (datasets, fill values and attributes, every
shareable message type, libver earliest and latest).

Regression test: sohm_shared_messages_resolve (h5py writes files sharing
each message type on its own and all of them; values and attributes
checked).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-25 22:08:45 -05:00
osobhandClaude Opus 5.5 90e050944f fix(format): refuse a local heap whose free list leaves the heap 
libhdf5 walks a local heap's free list when it loads the heap's data and
refuses the heap ("bad heap free list") when a free block starts or ends
outside the data segment, or links to offset 0. We never looked at the
free list, so a damaged old-style group listed names read from the broken
heap: once the user block of cve-2021-36977.h5 was applied, its root
listed eight garbage names where libhdf5 fails.

LocalHeap::validate_free_list (new) mirrors H5HL__fl_deserialize, with a
cycle bound, and accepts H5HL_FREE_NULL (1) or an all-ones head as the
end of the list. Like libhdf5 it runs when the first name is needed, not
on parse, so an empty group with a damaged heap still lists as empty
(cve-2018-13871.h5, cve-2024-29166.h5, gh-4431-poc-03.h5 keep matching
h5py).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-25 22:07:50 -05:00
osobhandClaude Opus 5.5 a6e90f3ee3 fix(format): apply the base address of files with a user block 
A file may start with a user block (h5py userblock_size, h5jam), putting
the superblock at 512, 1024, ...; every address in the file is then
relative to the superblock. The signature search found it, but every
reader passed the whole file to the parsers, so addresses landed
userblock bytes early and the root group failed with
InvalidObjectHeaderVersion (twithub.h5, twithub513.h5,
h5clear_fsm_persist_user_*.h5).

Readers now view the file from the superblock on, taking the signature's
position as the base address as libhdf5 does: File (mmap, buffered,
from_bytes), MmapFile, LazyFile, AsyncHDF5File, the VOL and MPI VOL
readers, the HNSW loader and external VDS source files. File, MmapFile
and LazyFile gain user_block_size(). The new signature::split_user_block
returns the two parts, and Superblock::parse refuses a non-zero offset
(UserBlockNotStripped) so a format-level caller cannot silently apply
superblock-relative addresses to the whole file.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-25 22:07:50 -05:00
osobhandClaude Opus 5.5 0555794850 fix(format): read version-1 shared message addresses after the heap offset 
A version-1 shared message (HDF5 1.6) embeds the target as a symbol-table
entry: after six reserved bytes comes a length-sized local-heap offset,
then the object header address. We read the heap offset as the address,
so datasets using a committed datatype in 1.6-era files (tcompound.h5,
tcompound2.h5) failed with InvalidObjectHeaderVersion. parse_shared_ref
now takes length_size and skips the offset, as libhdf5 does.

Resolving a reference also no longer falls back to the first message of
any type in the target header: a missing target message is
SharedMessageTargetMissing instead of garbage.

Fixture: tcompound.h5 from libhdf5's tools/test/testfiles (8 KiB).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-25 22:07:50 -05:00
osobhandClaude Opus 5.5 efc2dc53c9 fix(format): read array members of version-1 compound datatypes 
HDF5 1.6 encoded a compound member that is a fixed-size array through
legacy per-member fields (dimensionality, permutation, four dimension
sizes) that the v1 decoder skipped, so a [4] i32 member read as one i32
with the wrong size. Build the array type from those fields as libhdf5
does (ignoring the permutation) and refuse more than four dimensions.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-25 22:07:50 -05:00
osobhandClaude Opus 5.5 5c2f656fe7 docs: first conformance report and baseline (42b81d9, tank)
CI / test-arm64 (pull_request) Successful in 1m6s
Details
CI / test (pull_request) Successful in 5m0s
Details 
467 of 697 files read identically to h5py 3.16 / HDF5 2.0, 123 our-error,
15 mismatch (2 an h5py big-endian VL bug), 92 libhdf5 cannot read; no
panics, hangs, crashes or OOM.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-25 22:06:00 -05:00
osobhandClaude Opus 5.5 945b13a1f1 ci: nightly conformance sweep 
Runs conformance/run.sh in rust:latest on a schedule and on demand, with its
own venv (pinned h5py/numpy/hdf5plugin/netCDF4) and hdf5-tools. Fails on any
panic, hang, crash or OOM in clawhdf5 and on a drop against
conformance/baseline.json; prints CONFORMANCE.md into the job log and
uploads nothing. Plain git checkout, no JavaScript actions.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-25 22:06:00 -05:00
osobhandClaude Opus 5.5 9179aa356e feat(conformance): in-repo, reproducible conformance sweep 
conformance/run.sh fetches eight public HDF5 corpora pinned by commit
(conformance/corpus.txt) into a gitignored cache, reads every file with
clawhdf5 (conformance/probe, a crate outside the workspace) and with
h5py/libhdf5 (ref.py), and the CVE files with h5dump, each under a timeout
and an address-space limit; compare.py classifies the files, report.py
writes CONFORMANCE.md and check.py gates on panics/hangs/crashes/OOM and on
regressions against conformance/baseline.json. ~25 s once cached.

Changes from the ad-hoc audit harness:
- the probe compares non-IEEE-layout floats (N-Bit) and integers with a bit
  offset or reduced precision as the values libhdf5 converts them to, not
  raw file bytes: 8 files that showed as mismatches now read identically;
- ref.py exits without tearing down h5py objects: libhdf5 2.0 aborts while
  freeing them for two files about half the time, which flipped them
  between ok and h5py-cannot-read from run to run;
- the file list is defined (list_files.py): netCDF classic files are left
  out, 11 HDF5 files the ad-hoc sweep missed are in.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-25 22:06:00 -05:00
osobhandClaude Opus 5.5 e94a52a88b fix(format): read unmapped VDS elements as the virtual dataset's fill value 
Elements of a virtual dataset that no mapping supplies (unmapped regions,
a missing source file, a missing source dataset) read as 0 instead of the
fill value libhdf5 returns — silent wrong data for any VDS created with a
non-zero fillvalue (read-matrix cases 0471/0472: -1 and 7 read as 0). A
missing source dataset was an error; libhdf5 reads it as fill.

Move VDS assembly into a new vds module following H5Dvirtual.c:
vds::read_virtual_dataset takes the dataset's fill value and a
VdsFileResolver that can refuse a name, and reports how many elements were
unmapped. Sources are read with their own fill value, and a source whose
datatype differs from the virtual dataset's is an error (libhdf5 converts).
File passes the dataset's fill value, resolves source names against the
virtual file's directory, and refuses names that leave it with an error
instead of reading them as fill. read_selection on a VDS goes through the
same fill-aware path.

The raw-read API (read_raw_data_full*) has no fill value, so it now errors
for a VDS with unmapped elements instead of guessing zeros.

Tests: vds_interop::vds_unmapped_regions_read_as_fill_value (external,
same-file, missing file/dataset, sparse source with its own fill, int
fill; earliest and latest format) and
vds_source_outside_directory_is_an_error_not_fill, both against h5py;
integration_test::v4_virtual_dataset_raw_api_refuses_to_guess_the_fill_value.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-25 22:03:47 -05:00
osobhandClaude Opus 5.5 d54a0f4737 feat(format): read the other attributes when one cannot be read 
attrs() read every attribute of an object through extract_attributes_full,
so one attribute it could not read (a corrupt or unsupported attribute
message, or a heap object it could not locate) failed all of them — the
same shape as the huge-object bug, where one 8 KiB attribute hid every
attribute on a NetCDF file's root group.

- clawhdf5-format: new attribute::extract_attributes_tolerant returns the
  attributes it could read plus one error per attribute it could not.
  Errors in the attribute index itself (Attribute Info message, dense
  heap header, B-tree) still fail, since then it is unknown which
  attributes exist. extract_attributes_full is unchanged (strict); both
  share one implementation.
- clawhdf5: attrs() on Group/Dataset, MmapGroup/MmapDataset and
  LazyGroup/LazyDataset leaves an unreadable attribute out (documented),
  and the new attrs_with_errors() returns the map with the per-attribute
  errors. A value is either returned complete or not at all.

Regression test: one_unreadable_attribute_does_not_hide_the_others (h5py
writes 11 dense attributes; one message's version byte is corrupted;
before: attrs() failed with InvalidAttributeVersion(127), after: the 10
others come back with their values and one error is reported).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-25 22:03:09 -05:00
osobhandClaude Opus 5.5 aadfd18d4c fix(format): list soft links as their targets, like h5py 
Group::datasets()/groups() (and the Mmap/Lazy handles) listed only hard
links, so a soft link to a dataset or group was missing, and dataset(name)
/ group(name) on a group handle could not open one. In old-style (symbol
table) groups a soft link's entry has no object header address, and the
listing failed outright trying to parse one.

The three facade handles each had their own copy of the child-listing
code; they now share group_v2::resolve_group_children, which returns hard
links plus soft links resolved to their targets (relative targets from
the group holding the link, via the new resolve_path_from). A dangling or
cyclic soft link, an external link and a user-defined link are left out —
h5py lists their names but cannot open them. Any other error met while
resolving is returned, not hidden.

Path resolution now walks a relative soft link's target from the group
holding it instead of rebuilding the path from the root (same result,
one less re-walk), and ignores "." components.

Regression test: soft_links_are_listed_as_their_targets (h5py writes
absolute, relative, group, dangling, cyclic and external links with
libver latest and earliest; listings compared with h5py for File,
MmapFile and LazyFile).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-25 22:00:46 -05:00
osobhandClaude Opus 5.5 2c6c6c176e fix(format): decode the version-1 VDS mapping list HDF5 2.0 writes 
With a 2.0 low version bound, libhdf5 stores the VDS mapping list as heap
block version 1: every entry starts with a flags byte (0x04 same file, no
file name; 0x01/0x02 file/dataset name shared with an earlier entry, whose
index is stored in place of the name). The parser treated only a leading
0x04 byte as special, so a 0x00 flags byte read as an empty (same-file)
name and shared names were read as garbage.

Decode it as H5D__virtual_load_layout does, refusing unknown flags,
forward references and block versions above 1.

Test: vds_interop::vds_mapping_block_version1_shared_names (h5py
libver=("v200","v200") with repeated long names; failed before with
"unknown dataspace selection type") plus the exact heap block as a unit
test.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-25 21:59:15 -05:00
osobhandClaude Opus 5.5 190918a478 feat(format): decode hyperslab selection versions 1 and 2 in VDS mappings 
libhdf5 serializes a VDS hyperslab as version 1 (irregular, 4-byte block
corners) for the default format bounds, and as version 2 (regular, 8-byte)
for unlimited selections in the 1.10 format. Only version 3 was accepted,
so every h5py VDS written with default libver failed with "only version-3
hyperslab selections are supported" (5 libhdf5 test files in the sweep).

Decode all three versions following H5S__hyper_deserialize, including
irregular hyperslabs (a union of blocks, enumerated in row-major order as
libhdf5 iterates them) and the all-ones "unlimited" count/block marker.
SerializedSelection exposes the raw form for unlimited-mapping support.

Test: vds_interop::vds_version1_irregular_hyperslab_selections compares
default-libver h5py VDS reads (contiguous, strided and 2-D block mappings)
with libhdf5's values.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-25 21:57:32 -05:00
osobhandClaude Opus 5.5 38d0d4de02 fix(format): skip user-defined links instead of failing the group 
Link types 65-255 are user-defined: their target is only meaningful to
the application that registered the link class. LinkMessage::parse
rejects them with InvalidLinkType, and group traversal propagated that,
so one such link made the whole group unlistable and every path through
it unresolvable (libhdf5's tall.h5 and tudlink.h5, class 187).

Group traversal (compact and dense) now leaves user-defined links out,
the way h5py leaves out links it cannot open; reserved types (2-63) are
still an error.

Regression test: user_defined_links_do_not_break_the_listing, on
libhdf5's own tools/test/testfiles tall.h5 and tudlink.h5 (BSD-style
HDF5 licence, 10 KB and 1 KB), committed as fixtures.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-25 21:57:18 -05:00
osobhandClaude Opus 5.5 1c85986079 fix(format): read huge, tiny and filtered fractal heap objects 
A heap ID's type is in bits 4-5 of its first byte (H5HF_ID_TYPE_MASK
0x30); bits 6-7 are the ID version. The reader took the type from bits
6-7, so every huge object ID (0x10) was decoded as a managed one and
failed — and since dense attributes are read all at once, one attribute
over the heap's 4 KiB managed limit made every attribute on its object
unreadable (netcdf4-python's issue671.nc / issue672.nc).

- Huge objects (type 1): located directly from the ID when address and
  length fit in it, otherwise through the huge-object v2 B-tree (record
  types 1 and 2); filtered huge objects are decoded with the heap's
  pipeline and their filter mask.
- Tiny objects (type 2): read from the ID itself.
- Filtered heaps: the header's pipeline is parsed (it was skipped short,
  so the header checksum was read from the wrong place), indirect-block
  entries for direct blocks carry their filtered size and mask, and
  direct blocks are decoded before objects are read from them.
- An unknown ID version is an error.

FractalHeapHeader gains huge_btree_address, filter_pipeline,
root_direct_block_filtered_size, root_direct_block_filter_mask,
offset_size and length_size; read_managed_object now accepts any ID type.

Regression tests (h5py-written, compared with h5py):
dense_attribute_stored_as_a_huge_heap_object, dense_group_with_a_huge_link,
dense_group_with_a_filtered_link_heap; unit tests
tiny_object_is_read_from_the_id, huge_object_with_a_direct_id,
unknown_heap_id_version_is_refused.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-25 21:55:51 -05:00
osobhandClaude Opus 5.5 c7092722aa fix(format): locate the address in version-1 shared messages 
A version-1 shared message reference is version, type, six reserved bytes
and then an old-style symbol table entry: link-name offset (length size),
object header address, cache type, reserved, scratch. We read the address
straight after the reserved bytes, i.e. the link-name offset, and the
committed datatype lookup failed with InvalidObjectHeaderVersion (the bytes
checked in tcompound.h5: name offset 0x10, then 0x590 = /type1). Datasets
of 1.4/1.6-era files that use a committed datatype were unreadable.

Skip the name offset. parse_shared_ref has no length size, so add
parse_shared_ref_sized and use it in every internal caller;
parse_shared_ref keeps its signature and assumes length size == offset
size. The old parse_v1_ref unit test encoded the wrong layout and now uses
the real bytes.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-25 21:55:31 -05:00
osobhandClaude Opus 5.5 36356ba8a1 fix(format): keep the array dimensions of compound v1 members 
Compound datatype version 1 carries, per member, a dimensionality and four
dimension sizes (HDF5 before 1.4 had no array class). The parser skipped
those 28 bytes, so a member such as `f: f32[4]` came back as a single f32
at the member's offset: the compound's size was right but its members were
wrong. libhdf5 wraps such a member in an array type of the first
`dimensionality` sizes and ignores the permutation; do the same, and
reject a dimensionality above 4 as libhdf5 does.

Only files old enough to also use layout message v1 have these, so this
became reachable with the previous commit (tarrold.h5, tcompound.h5).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-25 21:55:31 -05:00
osobhandClaude Opus 5.5 85eb7f5ce2 feat(format): read Data Layout message versions 1 and 2 
HDF5 1.4/1.6-era files store the layout as version 1 or 2: version,
dimensionality, class, 5 reserved bytes, an address (contiguous and chunked
only), dimensionality 32-bit sizes (with the trailing element-size
dimension) and, for compact storage, a 32-bit size and the raw data. They
failed with InvalidLayoutVersion — 84 of the 686 files in the audit sweep,
205 datasets.

Map them onto the existing variants: chunked uses the same version-1
B-tree chunk index as version 3 and is reported as version 3, so every
chunked read path (filters, selections, caches) applies unchanged.
Contiguous size is the product of the stored dimensions, which is what
libhdf5 computes from the dataspace; a disagreement fails the reader's size
check instead of returning wrong data.

Fixtures are HDF5's own deflate.h5 (v1, chunked + deflate) and
h5ex_g_iterate.h5 (v2, contiguous, one unallocated dataset); the new
interop test compares every dataset byte for byte against h5py.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-25 21:55:31 -05:00
osobhandClaude Opus 5.5 8196fab72a fix(format): read v2 B-tree internal nodes with libhdf5's pointer widths 
An internal node's child pointer is an address, the child's record count
and (below the first internal level) the child subtree's total record
count. libhdf5 (H5B2__hdr_init) encodes the record count in the width of
a leaf's maximum and the subtree total in the width of cum_max_nrec for
that depth, computed level by level from the node size. The reader
guessed 2 * leaf_max and leaf_max^depth, which agree at depth 2 but not
at depth 3: a 24 000-link group's name index has depth 3, its root's
pointers were read 3 bytes wide instead of 2, and listing failed with a
garbage heap offset.

Regression tests: dense_group_with_a_three_level_name_index (h5py writes
24 000 links; listing compared with h5py) and
subtree_capacity_matches_libhdf5.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-25 21:53:00 -05:00
osobhandClaude Opus 5.5 8ebd488d9e fix(format): size fractal heap child indirect blocks by their row's span 
A child indirect block in row r of a fractal heap's doubling table spans
that row's block size of heap space, so it has
log2(size) - log2(start_block_size * width) + 1 rows (libhdf5's
H5HF__dtable_size_to_rows). The reader used row - first_indirect_row + 1,
which undercounts, so every object stored past the root block's direct
rows (512 KiB with libhdf5's defaults) was unreachable: dense groups with
a few thousand long link names, or ~20 000 short ones, could not be listed.

Regression test: dense_group_whose_heap_outgrows_the_root_direct_rows
(h5py writes 2 500 links with 248-byte names; listing compared with h5py).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-25 21:52:07 -05:00
osobh 42b81d9f1c Merge pull request 'Fix silent wrong data and libhdf5 interop found by the HDF5 audit' (#11) from fix/phase0-correctness into main
CI / test-arm64 (push) Successful in 1m7s
Details
CI / test (push) Successful in 5m50s
Details 
Reviewed-on: #11
 2026-09-26 02:42:53 +00:00
osobhandClaude Opus 5.5 72b9cfb1e1 docs: record the 2026-09-25 HDF5 audit fixes and open gaps
CI / test-arm64 (pull_request) Successful in 1m21s
Details
CI / test (pull_request) Successful in 5m59s
Details 
CHANGELOG: upgrade notes (changed read results for max-shape files,
saturating conversions, new writer errors, format-crate API changes) and
the reader/writer correctness fixes. known-issues: the silent-wrong-data
table with before/after sweep numbers, the gaps still open, and a
correction to the Extensible Array entry, which said files we wrote were
unaffected. CLAUDE.md: clawhdf5-gpu is vector distance computation, not
I/O, and clawhdf5-filters holds only deflate backends (no Blosc).

Also a facade test that libhdf5's 20-bit N-Bit float test data reads as
libhdf5's values.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-25 21:26:56 -05:00
osobhandClaude Opus 5.5 650f355219 ci: run the hdf5plugin LZ4/Zstd interop tests 
The interop step built writer_h5py_tests without the lz4/zstd features,
so the hdf5plugin round-trips added with the registered LZ4 framing and
the Zstd content-size fix never compiled in CI, and CI never installed
hdf5plugin.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-25 21:24:50 -05:00
osobh 7f5cfee281 Merge branch 'fix/p0-filters' into fix/phase0-correctness 
# Conflicts:
#	crates/clawhdf5-format/src/filters.rs
 2026-09-25 21:21:19 -05:00
osobh c5302e587e Merge branch 'fix/p0-writer-meta' into fix/phase0-correctness 2026-09-25 21:20:59 -05:00
osobh e1115bc92a Merge branch 'fix/p0-reader-numeric' into fix/phase0-correctness 2026-09-25 21:20:59 -05:00
osobh 36d7a6f234 Merge branch 'fix/p0-chunked-read' into fix/phase0-correctness 2026-09-25 21:20:59 -05:00
osobh 4b23ad697c Merge branch 'fix/p0-chunk-index' into fix/phase0-correctness 2026-09-25 21:20:59 -05:00
osobhandClaude Opus 5.5 e7f2d8575d fix(format): import format! for the no_std chunk index planner 
The maxshape checks added to chunked_write use format!, which a no_std
build has to import from alloc (scripts/check-nostd.sh).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-25 21:18:13 -05:00
osobhandClaude Opus 5.5 7c1968a34a fix(format): resolve shared fill value messages instead of zero-filling 
dataset_fill_value treated a shared Fill Value message as "no fill
value", so unwritten storage of a dataset whose fill value lives in the
file's shared-message (SOHM) heap read as zeros rather than its fill
value. libhdf5 shares fill values whenever the file has a SOHM index for
them.

- fill_value::dataset_fill_value_in follows the reference (another object
  header, or the SOHM heap); read_full_with_fill and the facade's
  selection read use it.
- dataset_fill_value, which has no file to follow a reference into, now
  returns UnresolvedSharedMessage for a shared message instead of None.
- shared_message::load_sohm_table / message_data_with_sohm load the SOHM
  table from the superblock extension on demand.
- parse_sohm_table skipped each index's leading version byte, reading
  every field one byte off; SOHM references could never resolve.

Fixture shared_fill_value.h5 (HDF5 2.0, gen_shared_fill.py): sohm_b read
[0,1,2,3,0,0,0,0] and now reads [0,1,2,3,-7,-7,-7,-7], as h5py does.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-25 21:18:12 -05:00
osobhandClaude Opus 5.5 6db13c60b8 docs: changelog for the filter interop fixes 
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-25 21:17:08 -05:00
osobhandClaude Opus 5.5 d99426be94 fix(format): allow Fletcher32 ahead of a compressor in the pipeline 
libhdf5 applies filters in pipeline order, so with Fletcher32 before
deflate (h5repack_filters.h5 /dset_all: shuffle, fletcher32, deflate; or
h5py's set_fletcher32() then set_deflate()) the compressor holds the
chunk plus a 4-byte checksum. decompress_chunk bounded every stage by the
chunk size and rejected it: "deflate: output exceeds size limit". Bound
each stage by the chunk size plus 4 bytes per Fletcher32 that precedes
it in the pipeline.

Test: fletcher32_before_deflate_decodes (h5py-written chunk, and our own
shuffle + fletcher32 + deflate round trip); failed before.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-25 21:16:48 -05:00
osobhandClaude Opus 5.5 f5505fb03d fix(format): keep maxshape == shape datasets contiguous 
Any maxshape forced chunked storage, even one equal to the shape, which
cannot grow. h5py and the library store such a dataset contiguously; we
now do too unless chunks (or a filter) are requested.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-25 21:15:55 -05:00
osobhandClaude Opus 5.5 95dcb04454 fix(format): scale-offset float decode with libhdf5's arithmetic 
D-scale floats were rebuilt as `minval + code / 10^D` in f64 and then
rounded to f32 once, but libhdf5 (H5Z_scaleoffset_modify_3/4 with
`float`/`powf`) computes `(float)(int)code / powf(10, D) + min` in single
precision. The two differ by 1 ULP for some values: le_data.h5
/Scale_offset_float_data_{le,be} gave 1.6663332 (0x3fd54a69) where
libhdf5 gives 1.6663333 (0x3fd54a6a). Use f32 arithmetic for 4-byte
floats and `(double)(long)code / pow(10, D) + min` for 8-byte ones.

Test: scaleoffset_float_dscale_matches_libhdf5_bits (le_data.h5 float
LE/BE and double chunks, bit-exact against h5py); failed before.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-25 21:15:49 -05:00
osobhandClaude Opus 5.5 1dba7b465a fix(format): index datasets with several unlimited dims by B-tree v2 
A dataset with more than one unlimited dimension got an Extensible Array
index, which libhdf5 refuses ("already found unlimited dimension"), so
the whole file failed to open in h5py and h5dump. The previous commit
turned that into a write error; this one writes what the library itself
uses there: a version-2 B-tree chunk index (record type 10/11), as a
single leaf of the library's 2048-byte node size, or a larger leaf when
the records do not fit. The root's record count is 16-bit, so more than
65535 chunks is still refused rather than written wrong.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-25 21:14:47 -05:00
osobhandClaude Opus 5.5 57e938c438 fix(format): honour unknown-message flags the way libhdf5 does 
The object header parser failed on an unknown message with flag bit 3
set and ignored bit 7. Per the spec, bit 3 means "fail if unknown and
the file is opened for writing" and bit 7 "fail if unknown, always".
The parser only reads, so it now ignores bit 3 (as libhdf5 does for a
read-only open) and refuses bit 7, in v1 headers, v2 headers and their
continuation chunks.

On libhdf5's conformance file tbogus.h5 (added as a fixture) we used to
refuse Dataset2 and open Dataset3; we now match libhdf5: Dataset1, 2, 4
and 5 open, Dataset3 is refused.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-25 21:14:30 -05:00
osobhandClaude Opus 5.5 3000b40cf3 fix(format): N-Bit pass-through flag and no-op (enum) members 
- libhdf5 sets cd_values[1] ("need not compress") when every field is
  already full width and then stores the chunk unchanged
  (H5Z__filter_nbit: `if (cd_values[1]) HGOTO_DONE`). We ignored it and
  tried to unpack, so tfilters.h5 / h5stat_filters.h5 `/all` (shuffle +
  szip + deflate + fletcher32 + N-Bit) failed with "nbit: packed data too
  short". A type with no N-Bit parameters (cd = [3, 1, nelmts]) is now
  accepted the same way.
- Class 4 (H5Z_NBIT_NOOPTYPE: enum, string, opaque, ... members) is
  stored whole, 8 bits per byte; it was UnsupportedFilter(5)
  (h5repack_nested_8bit_enum_deflated.h5).

N-Bit on floats was not wrong in the filter: for le_data.h5 /
Nbit_float_data_* our output equals libhdf5's decoded bytes in the file
datatype (a 20-bit float, offset 7, bias 31). h5py's values differ
because libhdf5 then converts that custom float layout to IEEE, which
our datatype reader does not do; nbit_float_matches_libhdf5_file_type_bytes
pins the filter output and the doc comment says where conversion belongs.

Tests: nbit_need_not_compress_is_passthrough,
nbit_in_multi_filter_pipeline_matches_libhdf5 (tfilters.h5 chunk, szip
feature), nbit_compound_with_enum_member_matches_libhdf5 all failed
before; nbit_float_matches_libhdf5_file_type_bytes (guard).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-25 21:14:14 -05:00
osobhandClaude Opus 5.5 bc820fbd8c fix(format): refuse path-like group and dataset names 
FileWriter writes the root group plus one level of groups; it has no way
to create intermediate groups. create_group("a/b") therefore stored a
single link literally named "a/b", which no HDF5 reader can resolve
(h5py: "component not found"). Nesting would mean restructuring the
writer's layout around a group tree, so for now finish() rejects any
group, dataset or external-link name that is empty, "." or contains '/'.
Attribute names may still contain '/'.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-25 21:13:08 -05:00
osobhandClaude Opus 5.5 9066d34eaa fix(format): key the shared chunk cache by dataset 
A File is Send + Sync and keeps one ChunkCache for all its datasets.
The cached readers bound that cache to "the current dataset" with
ensure_dataset(addr), then checked, built and read its index and its
decompressed chunks in separate lock acquisitions. Two threads reading
two chunked datasets interleaved those steps, so one could store its
chunk index under the other's binding, or get the other's decompressed
chunk for the same coordinate: wrong data, or an index-out-of-bounds
panic when the ranks differed (16 threads x 40 reads over 24 datasets
panicked on every run).

The cache now keeps per-dataset state keyed by chunk-index address:
the chunk index, ChunkIndex and ChunkLayout per dataset (held as Arcs,
built outside the lock, first writer wins), and decompressed chunks
keyed by (address, coordinate). The chunked readers use the new
addr-taking methods (chunks_for, chunk_layout_for, get/put_decompressed_in,
prefetch_hint_in) exclusively. Memory stays bounded: decompressed data by
the existing byte/slot budget across datasets, indexes by at most 64
datasets and 2^20 index entries in total, dropping the least recently
used dataset's index first. Switching datasets no longer throws away the
other datasets' cached chunks.

The address-less methods remain and act on the dataset last bound with
ensure_dataset; they are documented as not for concurrent readers.

Regression: threads_reading_different_datasets_get_their_own_chunks
(crates/clawhdf5/tests/concurrent_chunk_cache.rs), plus cache unit tests
datasets_sharing_coordinates_stay_separate, dataset_indexes_are_bounded
and concurrent_readers_of_different_datasets_see_their_own_chunks.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-25 21:12:52 -05:00
osobhandClaude Opus 5.5 540fa08907 fix(format): write chunk indexes over the max extent, swizzled for EA 
The writer indexed chunks by their position in the current shape, the
same mistake the reader had. With a finite maxshape larger than the shape
the Fixed Array was sized for the shape, so libhdf5 looked up chunks past
its end ("addr overflow"); with the unlimited dimension anywhere but first,
e.g. maxshape (20, None), libhdf5 swizzles that dimension to the slowest
position and read our Extensible Array scrambled. Two unlimited dimensions
produced a file libhdf5 refused to open ("already found unlimited
dimension").

Chunks are now placed with the shared chunk_grid linearisation: Fixed
Array slots cover every chunk of the maximum extent (unwritten ones
undefined), Extensible Array indexes are swizzled, Single Chunk is only
used when the maximum extent is one chunk, and a maxshape that is smaller
than the shape, has more than one unlimited dimension, or would need an
absurd Fixed Array is an error instead of a bad file.
build_chunked_data_from_precompressed now returns a Result.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-25 21:12:47 -05:00
osobhandClaude Opus 5.5 14876b8ae5 fix(format): give empty string attributes a 1-byte type 
An empty AttrValue::String (or a StringArray of empty strings) was
written with a size-0 fixed-length string type. libhdf5 rejects that
("invalid datatype size"), and the failure takes every attribute on the
object with it. Strings are now at least 1 byte, NUL-padded, which is
how h5py stores "" and reads back as "" in both h5py and our reader.
check_encodable also refuses a size-0 string type passed in directly.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-25 21:12:12 -05:00
osobhandClaude Opus 5.5 5935e13866 fix(format): decode SZIP chunks the way libhdf5 does 
SZIP-filtered datasets from libhdf5 came back as garbage or zeros with no
error (ref_szip.h5, h5repack_szip.h5, noencoder.h5, le_data/be_data
Szip_float_data_*), and 64-bit ones failed with "invalid bits per
sample" (h5wasm compressed.h5). The decoder called aec_buffer_decode
directly, but libhdf5 goes through szlib's SZ_BufftoBuffDecompress
(H5Zszip.c), which libaec implements with reshaping (sz_compat.c).
Differences, all fixed:

- H5Zszip.c prefixes the stream with the 4-byte LE uncompressed size; it
  was fed to libaec as data.
- 32- and 64-bit samples are coded as byte planes of 8-bit samples and
  must be de-interleaved.
- The reference sample interval is ceil(pixels_per_scanline /
  pixels_per_block), not a fixed 128.
- Scanlines that are not a whole number of blocks are padded and must be
  unpadded.
- Byte order comes from the MSB option bit; LE data was decoded as MSB.

Test: szip_decodes_libhdf5_chunks_exactly compares chunks from HDF Group
test files (noencoder.h5, le_data.h5) and an h5py-written file (64-bit,
16-bit, padded scanlines, NN and EC) byte for byte with h5py's values;
it failed before on the first case.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-25 21:11:24 -05:00
osobhandClaude Opus 5.5 8c3ef996ea fix(format): write fill times with libhdf5's codes; add fill values 
FillTime::to_byte had the fill-time field rotated against libhdf5
(H5D_FILL_TIME_ALLOC = 0, NEVER = 1, IFSET = 2): Never was written as
ALLOC, Alloc as IFSET and IfSet as NEVER, as h5py reported. The flags
byte is now late allocation plus the right code, and FillTime::from_byte
decodes it.

The default becomes IfSet, which is libhdf5's default and exactly the
byte (0x0a) every dataset was already written with, so default output
does not change; `Alloc` was documented as the C library's default but
never was. DatasetCreateProps follows.

DatasetBuilder::with_fill_value sets a user-defined fill value (one
element's stored bytes, checked against the datatype size), written as a
defined value in the fill value message. h5py reports it, and extending
the dataset in h5py fills the new elements with it.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-25 21:11:07 -05:00
osobhandClaude Opus 5.5 74fdf0582b fix(format): write paged files libhdf5 can open 
FileWriter::with_page_size wrote a "version 4" superblock with an extra
page-size field. HDF5 has no superblock version 4, so libhdf5 refused
every such file ("bad superblock version number").

A paged file is now what libhdf5 itself writes for fs_strategy="page":
a v3 superblock whose extension object header holds a File Space Info
message (strategy PAGE, the page size, free space not persisted; same
bytes and flags as HDF5 2.0), with the file padded to a whole page.
h5py opens it, reports the strategy and page size, and can modify it in
r+ mode. Page sizes outside libhdf5's 512 B..1 GiB are an error.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-25 21:09:31 -05:00
osobhandClaude Opus 5.5 44f5f8b5c5 fix(format): index every Extensible Array chunk, not just the first 244 
The Extensible Array writer only filled the index block's 4 inline
elements and the 6 data blocks it addresses directly (240 elements); its
super block addresses were always undefined. Chunks from index 244 on were
written to the file but never indexed, so they read back as fill values in
our reader and in libhdf5, without an error.

The writer now lays out data blocks and super blocks for any element
count as H5EA__hdr_init sizes them, pages data blocks larger than 1024
elements (page-init bits in the owning super block), leaves blocks with no
defined element unallocated, and records real header statistics
(max_idx_set is one past the highest defined index).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-25 21:09:29 -05:00
osobhandClaude Opus 5.5 2f252df084 fix(format): return whole VL sequences from read_vl_bytes 
read_vl_bytes cut each element to the reference's length field, which
counts sequence elements, not bytes: a VL int32 [1, 2, 3] came back as
3 bytes. Return the whole global-heap object, which is element count x
base size bytes. No in-tree caller depended on the old behaviour.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-25 21:08:37 -05:00
osobhandClaude Opus 5.5 c8c2930fc0 fix(format): read enum and bool datasets through their base integer type 
read_i64/read_u64/read_i32/read_f64/read_f32 refused enumeration
datatypes, including h5py's bool (an enum of int8), with a type
mismatch. Read them as their base type's integer values, the way array
datatypes already read through theirs.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-25 21:08:37 -05:00
osobhandClaude Opus 5.5 417c9516ca fix(format): decode floats by their datatype fields, not their size 
Every 2-byte float was decoded as IEEE half, so bfloat16 (HDF5 2.0's
H5T_FLOAT_BFLOAT16*, or any custom 8-bit-exponent type) read wrong:
1.5 as 1.9375, +inf as NaN. 1-byte FP8 floats were refused.

Read the exponent/mantissa location and size and the bias from the
datatype message: IEEE half/single/double keep their existing paths
(half still through clawhdf5_format::float16), any other IEEE-style
layout up to 64 bits whose values fit f64 (bfloat16, FP8 E4M3/E5M2, ...)
is decoded generically, and the bulk-copy and zero-copy fast paths now
require the IEEE layout rather than just the size. Datatypes with fields
that describe no float still fall back to IEEE by size; layouts that
cannot be represented in f64 (x87 80-bit, binary128) remain an error.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-25 21:08:37 -05:00
osobhandClaude Opus 5.5 53dbddb07b fix(format): saturate out-of-range integer reads instead of truncating 
Reading wider or differently-signed integers kept the low bits: i64
2^40+5 read as i32 was 5, u64::MAX read as i64 was -1, and -1 read as
u64 was 4294967295. u32 data read as i32 also took the bulk-copy fast
path meant for i32. Saturate at the target range like libhdf5's hard
conversions (a negative value read as unsigned is 0), and keep the i32
fast path to signed data.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-25 21:08:37 -05:00
osobhandClaude Opus 5.5 081341b433 fix(format): convert float data read as integers instead of returning bit patterns 
read_i32/read_i64/read_u64 on a floating-point dataset reinterpreted the
IEEE bits (1.5 read as i64 was 4609434218613702656). Convert like
libhdf5's hard conversions instead: truncate toward zero and saturate at
the target range; NaN reads as 0.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-25 21:08:37 -05:00
osobhandClaude Opus 5.5 d074385944 fix(format): read partial edge chunks stored unfiltered 
Layout message v4 flag bit 0 (H5D_CHUNK_DONT_FILTER_PARTIAL_CHUNKS, set
with H5Pset_chunk_opts) makes libhdf5 store every chunk that extends past
the dataset's extent without the filter pipeline, while its filter mask
still reads 0. The parser ignored the flag, so readers tried to inflate
raw bytes: libhdf5's own h5fc_edge_v3.h5 failed with "deflate: ...
unknown compression method".

DataLayout::Chunked gains dont_filter_partial_edge_chunks (always false
for v3), and list_chunks — the one place every read path gets its chunk
list from — marks such partial chunks as having skipped every filter, so
the full, cached, indexed, parallel and selection readers all copy them
as-is. chunked_write.rs gets `..` in one exhaustive test pattern for the
new field.

Regression: libhdf5_edge_chunk_fixture_reads (h5fc_edge_v3.h5 from the
HDF5 tools test files, committed as a 2.5 KB fixture), and
h5py_unfiltered_partial_edge_chunks_read (the flag set through h5py's
bundled libhdf5 via ctypes, as h5py has no binding for it: fixed array,
extensible array and B-tree v2 indexes, 1-D and 2-D, plus a hyperslab
of the last chunk), and v4_chunked_dont_filter_partial_edge_chunks_flag.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-25 21:08:36 -05:00
osobhandClaude Opus 5.5 4a1876faf2 fix(format): page Fixed Array data blocks past 1024 chunks 
The Fixed Array writer always packed every element into one data block
behind one checksum. Past 2^10 elements libhdf5 (and our reader) expect a
paged block: a page-init bitmap after the prefix, then one checksummed page
per 1024 elements. Any dataset with more than 1024 chunks and no unlimited
dimension failed with "incorrect metadata checksum" in h5py, h5dump and
our own reader.

build_fixed_array_at now takes one Option<WrittenChunk> per array slot so
later fixes can leave unallocated slots.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-25 21:07:08 -05:00
osobhandClaude Opus 5.5 be88e3fec7 fix(format): encode Time, BitField, Opaque and Reference datatypes 
Datatype::serialize returned an empty message for these four classes, so
any dataset or attribute of them (including a Raw attribute copied from
another file) was unreadable by libhdf5 ("ran off end of input buffer
while decoding"). They now encode exactly as libhdf5 does: legacy object
and region references as datatype version 1, H5T_STD_REF kinds as version
4 with their encoding version, opaque tags NUL-padded to 8 bytes.

Parsing an opaque tag now stops at its first NUL, so libhdf5's padding
no longer becomes part of the tag. Datatype::check_encodable rejects
what has no encoding (an opaque tag over 248 bytes); FileWriter::finish
calls it for every dataset and attribute type.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-25 21:06:51 -05:00
osobhandClaude Opus 5.5 e162c013fd fix(format): bound each filter stage by what the stages before it produce 
Every decode stage was capped at the chunk's decoded size. That holds
only when every filter ahead of the codec preserves size; Fletcher32
does not (it appends a 4-byte checksum), so a pipeline with Fletcher32
before deflate (NetCDF-4's fletcher32 -> shuffle -> deflate ordering,
h5repack's "all filters") failed with "deflate: output exceeds size
limit" on every chunk.

decompress_chunk_masked now computes each stage's bound by running the
chunk size forward through the filters that precede it in write order
(and that the chunk's mask did not skip): shuffle keeps the size,
Fletcher32 adds 4, any codec adds at most n/8 + 64. The cap is still a
small constant factor of the chunk, so a decompression bomb is rejected
as before (tested).

Shuffle also had to learn libhdf5's handling of a length that is not a
whole number of elements (chunk + checksum): shuffle the whole elements
and leave the trailing bytes in place, in both directions. It used to
refuse such data.

Regression: h5py_fletcher32_before_deflate_reads (fletcher->shuffle->
gzip, fletcher->gzip, shuffle->fletcher->gzip, and a 2-D i32 grid),
fletcher32_ahead_of_deflate_stays_bounded and
shuffle_leaves_a_partial_trailing_element_in_place.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-25 21:06:22 -05:00
osobhandClaude Opus 5.5 06dda26d85 fix(format): stop writing pcodec under Granular BitRound's filter ID 
Pcodec chunks were written as filter 32023, which the HDF Group registry
assigns to Granular BitRound (GBR). Pcodec has no registered ID (checked
2026-09-25 against hdf5_plugins/docs/RegisteredFilterPlugins.md, which
ends at 32033 with no pcodec entry). GBR's decode is a pass-through, so
libhdf5 with that plugin loaded would have returned the compressed bytes
as the dataset's values.

Write pcodec as 480, from the registry's testing/private range (256-511),
named "pcodec (clawhdf5 private)", and document it as non-interoperable:
only clawhdf5 with the `pcodec` feature reads it. Chunks under 32023 are
still read as pcodec when the filter is named exactly "pcodec" (what
clawhdf5 <= 2.7.0 wrote); any other 32023 is UnsupportedFilter.

Test: pcodec_uses_private_id_and_reads_legacy_32023.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-25 21:06:07 -05:00
osobhandClaude Opus 5.5 585e14d5e2 fix(format): honour each bit of a chunk's filter mask 
A chunk's filter mask has one bit per pipeline filter; bit i set means
filter i was not applied to that chunk (an optional filter that
declined, or a direct chunk write). Every read path treated any nonzero
mask as "no filters applied" and returned the stored bytes, so a chunk
that skipped only gzip in a shuffle+gzip pipeline came back still
shuffled (h5py write_direct_chunk with filter_mask=0b10: 8 of 32 values
wrong).

decompress_chunk_masked undoes the filters the mask leaves set and skips
the rest; an unsupported filter is no longer an error when the chunk
skipped it. The full, cached, sweep, indexed, parallel and selection
(partial_read) paths all use it, and a chunk is copied straight from the
file only when every filter was skipped. decompress_chunk is the mask-0
case.

Regression: h5py_partial_filter_mask_skips_only_masked_filters (1-D
shuffle+gzip with masks 0, 0b10 and 0b11; 2-D with 0b01; full and
hyperslab reads) and filter_mask_skips_only_the_masked_filters.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-25 21:05:07 -05:00
osobhandClaude Opus 5.5 183d96ee26 fix(format): record the content size in zstd frames 
Filter 32015 chunks were written with the streaming encoder
(zstd::encode_all), whose frames carry no content size. The registered
HDF5 Zstandard filter (H5Zzstd.c, libhdf5 + hdf5plugin) sizes its output
from ZSTD_getFrameContentSize and fails on such frames, so h5py could not
read our zstd datasets ("filter returned failure during read"). Compress
with the one-shot API, which records the size.

Tests: zstd_frames_record_content_size (content size was None before),
hdf5plugin_reads_our_zstd (ignored interop test; failed before).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-25 21:04:29 -05:00
osobhandClaude Opus 5.5 bba1560416 fix(format): lay Fixed/Extensible Array chunk indexes out by max dims 
Both indexes place each chunk at a linear index computed from the
dataset's maximum dimensions (libhdf5's max_down_chunks), and the
Extensible Array first swizzles its unlimited dimension to the slowest
position. We linearised by the current dimensions, so any dataset whose
shape was smaller than its maxshape, or whose unlimited dimension was not
the first, read back scrambled without an error: h5py libver="latest"
files with maxshape (10, None) or (20, 10), and the libhdf5 test files
h5fc_ext*.h5 and test_ld.h5.

The linearisation now lives in chunk_grid (shared with the writers), and
slots beyond the current extent are ignored as the library does.
read_fixed_array_chunks / read_extensible_array_chunks take the
dataspace's max dimensions.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-25 21:04:19 -05:00
osobhandClaude Opus 5.5 b36998ef01 fix(format): refuse object header messages over 64 KiB 
A v2 object header message has a 2-byte size field. The writer truncated
larger sizes to 16 bits, so an attribute over ~64 KiB (or a compact
dataset of 65532-65535 bytes, whose layout message adds 4 bytes) produced
a file libhdf5 rejects ("message of unshareable class flagged as
shareable", "bad flag combination").

ObjectHeaderWriter::serialize now returns a Result and fails on any message
over MAX_MESSAGE_SIZE; FileWriter::finish propagates it. Compact storage
falls back to contiguous above 65531 bytes, the real limit. Dense storage
for large attributes remains future work.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-25 21:04:07 -05:00
osobhandClaude Opus 5.5 aef8e766ae fix(format): write and read the registered HDF5 LZ4 filter format 
Filter 32004 chunks were framed as a 4-byte little-endian size plus one
LZ4 block. That is not the registered HDF5 LZ4 format (H5Zlz4.c: 8-byte
big-endian total size, 4-byte big-endian block size, then per block a
4-byte big-endian compressed length and the block, stored raw when the
length equals the block size), so libhdf5 + hdf5plugin could not read
our LZ4 datasets and we could not read theirs (h5ex_d_lz4.h5:
"lz4: 0 is not a valid match offset").

Write the registered format (cd_values[0] is honoured as the block size,
default 1 GiB like the plugin) and read it, multi-block and raw blocks
included. Chunks in the old framing stay readable: an HDF5 chunk is under
4 GiB, so a registered chunk always starts with four zero bytes and is at
least 12 bytes long, while an old one starts with four zero bytes only
when empty (5 bytes).

Tests: lz4_reads_registered_hdf5_format (chunk of the HDF Group's
h5ex_d_lz4.h5, block size 3), lz4_writes_registered_hdf5_format,
lz4_reads_legacy_clawhdf5_format, and hdf5plugin_reads_our_lz4 (ignored
interop test; failed before with "filter returned failure during read").

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-25 21:03:49 -05:00
osobhandClaude Opus 5.5 9ea44d473d fix(format): read v1 chunk B-tree key offsets as 8 bytes 
A type-1 (raw data chunk) B-tree key holds the chunk size, the filter
mask and one offset per dimension, and those offsets are always 8 bytes:
they are dataset coordinates, not file addresses. The reader used the
superblock's size-of-offsets for them, so in a file with 4-byte offsets
every key was misparsed. Unfiltered chunked datasets read as zeros (with
stray bytes where a misread address landed on data) and filtered ones
failed with "deflate: truncated stream".

Only the sibling and child addresses follow size-of-offsets now. The
unit-test B-tree builder wrote keys the same wrong way, which is why its
tests passed; it now matches the format.

Regression: h5py_four_byte_offsets_chunked_reads (h5py, set_sizes(4, 4)
and (4, 8); 1-D and 2-D, unfiltered and gzip) and the unit test
collect_chunks_with_four_byte_addresses.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-25 21:01:57 -05:00
osobhandClaude Opus 5.5 46203ea761 test(format): keep the 2026-09-20 B-tree v2 fuzz crash as a regression 
An 82-byte fuzz_btree_v2 crash input from 2026-09-20 was left untracked
in fuzz/artifacts. Replayed today it runs cleanly: the depth cap and
record budget added to B-tree v2 traversal that day fixed it. It is now
in the committed fuzz corpus, and a robustness test replays the fuzz
target's exact code path on it so a regression fails CI rather than
waiting for someone to run the fuzzer.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-25 20:38:34 -05:00
osobh 75bdb53342 Merge pull request 'Withdraw the ZeroClaw integration claims' (#10) from docs/withdraw-zeroclaw into main
CI / test-arm64 (push) Successful in 1m16s
Details
CI / test (push) Successful in 4m55s
Details 
Reviewed-on: #10
 2026-09-25 19:25:18 +00:00
osobhandClaude Opus 5.5 dd5b3f6633 docs: ClawBrainHub is the one verified consumer
CI / test-arm64 (pull_request) Successful in 1m3s
Details
CI / test (pull_request) Successful in 5m10s
Details 
The previous commit said clawhdf5 has no integration at all. ClawBrainHub
(clawverse/clawbrainhub) does use it: cbh-core reads and writes .brain
files through the facade, cbh-scanner uses the facade, and cbh-cli uses
clawhdf5_agent::bm25::BM25Index, all via path dependencies on this repo.
Checked on 2026-09-25 against main: it builds on its pinned toolchain and
its 204 tests pass. CLAUDE.md now records that, and that path
dependencies mean API changes here reach it directly.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-25 11:18:14 -05:00
osobh 79dfa78e8f Merge pull request 'Withdraw the OpenClaw integration claims' (#9) from docs/withdraw-openclaw into main
CI / test-arm64 (push) Successful in 1m18s
Details
CI / test (push) Successful in 5m40s
Details 
Reviewed-on: #9
 2026-09-25 16:14:36 +00:00
osobhandClaude Opus 5.5 87d64588e5 docs: withdraw the ZeroClaw integration claims
CI / test-arm64 (pull_request) Successful in 1m5s
Details
CI / test (pull_request) Successful in 5m34s
Details 
CLAUDE.md said ZeroClaw "imports this as a Cargo feature (clawhdf5
feature flag)" and uses clawhdf5 as its memory backend; the agent crate
called itself the "ZeroClaw agent memory HDF5 backend"; the migrator
claimed to read "the ZeroClaw layout". Checked on 2026-09-25 against
ZeroClaw v0.8.5 (its latest release), the osobh/zeroclaw fork (on
v0.8.5) and both histories back to February 2026:

- no `clawhdf5` feature, dependency or memory backend has ever existed
  in ZeroClaw; its backends are sqlite, lucid, postgres, qdrant,
  markdown and none, behind its own `Memory` trait;
- ZeroClaw's SQLite schema is a single `memories` table (id, key,
  content, category, embedding, created_at, updated_at); the
  migrator's memory_chunks/sessions/entities/relations layout never
  existed in ZeroClaw, so it cannot read a ZeroClaw database.

Decision: withdraw the claims (as with OpenClaw); clawhdf5 is a
standalone library with no framework integration. The migrator's
default layout is documented as its own. ZEROCLAW_VERSION keeps its name
and value (it is the persisted `edgehdf5_version` writer tag) with a
doc comment saying it is unrelated to ZeroClaw.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-25 11:08:41 -05:00
osobh bdadf3447c Merge pull request 'Ed25519-signed checkpoints; remove the no-op agent feature' (#8) from feat/signed-checkpoints into main
CI / test-arm64 (push) Successful in 1m16s
Details
CI / test (push) Successful in 5m42s
Details 
Reviewed-on: #8
 2026-09-25 15:58:19 +00:00
osobhandClaude Opus 5.5 0c65a27b00 docs: withdraw the OpenClaw integration claims
CI / test-arm64 (pull_request) Successful in 1m3s
Details
CI / test (pull_request) Successful in 5m48s
Details 
The docs described a "drop-in" OpenClaw memory backend enabled with
`memory.backend = "clawhdf5"`. Checked against OpenClaw's source and
docs (v2026.2.26 through v2026.9.6): that config was never valid —
v2026.2-v2026.7 accepted only "builtin"/"qmd" and rejected unknown
keys, so a Gateway given it refuses to start, and v2026.8.1 (OpenClaw
2.0) removed the key. No plugin was ever built (no manifest, no
registration, no tools), nothing was tested against OpenClaw, the
linked github.com/redclawsystems/openclaw is a 404, and
@redclaw/clawhdf5 was never published.

Decision (2026-09-25): not pursuing an OpenClaw plugin for now; ZeroClaw
is the integration target.

- Remove openclaw-integration.md, openclaw-config.md and
  migration-guide.md; add docs/openclaw.md: the status, what a memory
  plugin needs against v2026.9.6 (plugins.slots.memory, manifest with
  kind "memory", registerMemoryCapability / MemorySearchManager,
  prebuilt native packages), and what this repo has as building blocks.
- README, QUICKSTART, USE_CASES, ROADMAP (Track 7 withdrawn), CLAUDE.md
  and the `openclaw` module docs describe ClawhdfBackend as what it is:
  a Markdown-oriented library backend, not an OpenClaw plugin. The
  QUICKSTART example is corrected (the old one called a three-argument
  create that does not exist) and states its limits.
- packages/clawhdf5-node: marked unpublished and broken, "private": true
  so it cannot be published by accident; its bugs (snake_case vs
  camelCase fields, wrong addon path, no way to store an embedding,
  wrong WAL name) are recorded in docs/known-issues.md.
- Two broken rustdoc links fixed along the way.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-25 10:27:54 -05:00
osobhandClaude Opus 5.5 db9af7972c feat(agent): Ed25519-signed checkpoints
CI / test-arm64 (pull_request) Successful in 1m5s
Details
CI / test (pull_request) Successful in 4m50s
Details 
Makes the README's "cryptographically verifiable memory" true.

With HDF5Memory::set_signing_key(key), every checkpoint stores a signed
manifest of the store: a SHA-256 per memory record (text, embedding as
stored, channel, timestamp, session, tags, deleted flag, activation) in
a Merkle tree, plus hashes of the settings (and WAL mark), sessions and
knowledge graph. The signature, public key and manifest hashes go in
/meta; the per-record hashes in /integrity/record_hashes, so
HDF5Memory::verify(path, &public_key) can say which records changed, not
just that something did. A forged manifest fails the signature.

Decisions, as agreed:
- the key is set on the open store and never persisted;
- a signed store refuses to checkpoint without its key
  (MemoryError::SigningKeyRequired); remove_signature() is the
  deliberate way back to unsigned;
- checkpoints only: saves still in the WAL are not covered, and verify
  reports how many there are.

The hashes cover exactly what the file persists, in the form the loader
returns it (strings lose trailing NULs; an empty WAL mark is not
written), so untouched stores verify across any number of reopen and
checkpoint cycles. MemoryError becomes #[non_exhaustive] (it already
gains variants in this unreleased version).

CLI: keygen (owner-only key file), --signing-key / CLAWHDF5_SIGNING_KEY
on writing commands (create signs immediately), verify --public-key
(JSON; exit 2 if not valid), `signed` in create/stats output.

Tests: reopen/checkpoint cycles with awkward strings (f16 and f32),
refusal without the key, wrong and rotated keys, eight kinds of edit
each detected and located, a forged manifest, unsigned stores, NULs in
text, and an edit made in place with h5py that verify pinpoints.

Cost on tank (search_harness --signing-study --full, 3 runs): ~20% of a
checkpoint (+9 ms at 10K, +89-112 ms at 100K), verify 18.6 ms / 247 ms,
32 bytes per record in the file. New deps ed25519-dalek, sha2,
rand_core: pure Rust, the no-C check passes.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-25 10:13:34 -05:00
osobh 7706697feb Merge pull request 'Complete the consolidation benchmark: cheaper novelty scoring' (#7) from feat/consolidation-scaling into main
CI / test-arm64 (push) Successful in 1m7s
Details
CI / test (push) Successful in 5m28s
Details 
Reviewed-on: #7
 2026-09-25 15:05:58 +00:00
osobhandClaude Opus 5.5 4ecac65f22 chore(agent)!: remove the no-op agent feature 
It enabled nothing — the agent layer is always built — yet the README,
QUICKSTART and USE_CASES told people to pass it. Removed, with those
snippets fixed: they now depend on the git repository (nothing is on
crates.io, so `version = "2.0"` never resolved) and USE_CASES no longer
presents the `float16` feature as half-precision storage (that is
MemoryConfig::float16, on by default for new stores).

Breaking for anyone passing `features = ["agent"]`: drop it.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-25 09:53:53 -05:00
osobh c0f704c381 Merge pull request 'clawhdf5-migrate writes real agent stores; knowledge-graph fix; dated benchmark re-run' (#6) from feat/migrate-and-benchmarks into main
CI / test-arm64 (push) Successful in 1m19s
Details
CI / test (push) Successful in 5m34s
Details 
Reviewed-on: #6
 2026-09-25 14:52:35 +00:00
osobhandClaude Opus 5.5 00b0cb0035 perf(agent): cheaper novelty scoring; complete the consolidation benchmark
CI / test-arm64 (pull_request) Successful in 1m5s
Details
CI / test (pull_request) Successful in 5m39s
Details 
consolidation_efficiency never finished: stopped after 19 minutes on one
core while building its 100K case. Not the consolidation cycle (linear:
17 us at 100 records, 2.16 ms at 10K) but the setup — every add_memory
scores the new record's novelty against the whole working tier, the
benchmark lets that tier reach 50K, and each comparison recomputed both
norms: ~5e9 comparisons of three passes each.

ImportanceScorer::score_surprise now computes the new record's norm
once, takes each comparison in one fused, 8-lane pass (dot product and
the other norm together), and splits a working tier of 4096+ records
across threads with the `parallel` feature. Same results: tested against
the old cosine formula, including shorter, empty and zero vectors and
the parallel path. The work stays quadratic in the working-tier size by
design; with regular consolidation the tier stays near
working_capacity (100) and inserts are cheap.

The complete run takes 8 min 10 s on tank and fills in the 100K cycle
row (46.66 ms) and the memory-reduction table, which had never been
published. The binary no longer prints a record-count ratio as a
"BM25 Speedup" (never measured; Part 1 measures search latency) or
claims sub-linear cycle scaling (its own numbers grow slightly faster
than linearly).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-25 08:51:08 -05:00
osobh a7920bd4b3 Merge pull request 'Search options (source filters, re-ranking, confidence); float16 default' (#5) from feat/search-options into main
CI / test-arm64 (push) Successful in 1m8s
Details
CI / test (push) Successful in 5m40s
Details 
Reviewed-on: #5
 2026-09-25 12:44:15 +00:00
osobh 7e43b5366c Merge branch 'main' into feat/search-options
CI / test-arm64 (pull_request) Successful in 54s
Details
CI / test (pull_request) Successful in 4m35s
Details
2026-09-25 12:32:00 +00:00
osobhandClaude Opus 5.5 dce5559ff2 bench: re-run every stale BENCHMARKS.md section, dated and traced
CI / test-arm64 (pull_request) Successful in 54s
Details
CI / test (pull_request) Successful in 5m6s
Details 
Every undated or pre-September section re-run on one machine on one day
(tank, AMD Ryzen 7 7800X3D, 2026-09-24, commit 5c8323c), 24 commands run
serially with the load average checked before each, with the command
recorded for each section. A separate check traced every changed number
back to the raw output; its corrections are applied (e.g. the on-disk
~820 B/record is float16 plus always-deflated text on a synthetic corpus
of 40 distinct texts, not float16 alone).

Two apparent regressions were isolated rather than published:
- knowledge-graph traversal: a real bug, fixed in the previous commit;
- the write path: v2.3.0 built and run on the same machine measures the
  same as today, so the old 18 us / 6.17 ms figures (undated, other
  hardware) are not reproducible; float16 adds ~2 us per save and the
  int8 index nothing (both isolated by switching the bench's config).

Also:
- new multimodal_bench: cross-modal search at 1K/10K records, which the
  README claimed but nothing measured;
- footprint_bench reports whether it built float16 or f32 stores and
  takes --f32 (it kept printing "f32" after the default changed);
- README: performance tables, the "Why" table figures and the SQLite
  migration section (from the previous migrate commit);
- CHANGELOG for this branch.

Not re-run: consolidation_efficiency's 100K row and its memory-reduction
part (stopped for time), and cross_platform.sh.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-24 23:45:15 -05:00
osobhandClaude Opus 5.5 1b3bbb054a perf(agent): cache the knowledge graph's adjacency index 
bfs_neighbors and spreading_activation built an adjacency index over the
whole graph on every call (1efd82c), so a 2-hop BFS over 1K entities
paid to index every entity and relation first: 155 us, 6.5x the 24 us
the README quoted. Found by the dated benchmark re-run.

The index is now cached on KnowledgeCache and checked against a
fingerprint of the graph on each use — one pass over entity ids and
relation endpoints, no allocation — so any change, including direct
edits of the public entities/relations Vecs (schema.rs's load path
pushes to them), still triggers a rebuild. A test edits the graph
directly in every way (push, in-place rewire, pop + push at equal
length) between traversals.

tank, 2026-09-24: BFS 1K entities 155.1 -> 23.1 us, 100 entities
17.5 -> 5.23 us, spreading activation 100 22.8 -> 10.1 us.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-24 23:45:15 -05:00
osobhandClaude Opus 5.5 a8fb758489 fix(migrate): write a real clawhdf5-agent store 
clawhdf5-migrate wrote a layout of its own (/chunks, /sessions,
/entities, /relations, root attributes, no /meta or schema_version) that
HDF5Memory::open rejects, so a "migrated" SQLite database could not be
used as agent memory — contrary to the README.

It now writes through the agent's own API (HDF5Memory::create/open,
save_batch, the session cache and the knowledge graph), so there is no
second copy of the schema:

- sessions and entities/relations carry over; deleted rows become
  deleted records (or are left out with --skip-deleted);
- embeddings follow the library default (float16), --f32 opts out and
  --float16 is a hidden no-op, as in clawhdf5-cli; the `half`-based
  conversion is gone;
- every source row is checked before the output is created: a wrong
  embedding length, an empty embedding, a dimension that differs from
  an existing store's, or a float16 value beyond +-65504 is an error
  naming the chunk id, and an existing store is left untouched;
- --incremental opens the existing store, adds only rows it does not
  hold (matched by content) and follows the source's deleted flags;
- a source with no memory rows needs --embedding-dim;
- validation reads the result back with HDF5Memory::open_read_only,
  compares every field (embeddings bit for bit, round_to_f16 of the
  source for float16) and checks a migrated record is found by search.

clawhdf5-agent gains HDF5Memory::sessions()/sessions_mut(),
HDF5Memory::delete_batch (one save, all-or-nothing, no auto-compact),
SessionCache::add_at, and re-exports SessionCache/SessionEntry.

The old layout's per-dataset SHA-256 provenance attributes have no place
in the agent schema and are gone. An adversarial review found two
blockers (silent truncation of long embeddings; an --incremental
dimension check that could never fire) and four majors (a failed run
wiping the existing store, dim-0 stores, deleted-flag drift); all are
fixed with regression tests. 42 migrate tests, incl. h5py opening a
migrated store.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-24 23:45:15 -05:00
osobh 73bb068264 Merge pull request 'Files open in h5py again; float16 embedding storage' (#4) from feat/float16-embeddings into main
CI / test-arm64 (push) Successful in 1m17s
Details
CI / test (push) Successful in 5m10s
Details 
Reviewed-on: #4
 2026-09-25 03:18:07 +00:00
osobhandClaude Opus 5.5 5c8323cb1e feat(agent): new stores default to float16 embeddings
CI / test (pull_request) Successful in 5m27s
Details
CI / test-arm64 (pull_request) Successful in 1m9s
Details 
MemoryConfig::float16 now defaults to true for new stores, on
measurement: on the full LongMemEval haystack with real MiniLM
embeddings every retrieval metric matched f32 (previous commit), and at
100K the file is 48% smaller with faster checkpoints and opens.

Existing stores are unaffected: every agent store has recorded
`float16 = false` in /meta and keeps it. A test opens the v2.5.0
fixture, saves and checkpoints, and checks the embeddings are still f32
with the old rows bit-identical; another checks a new store is float16.

CLI: `create --f32` opts out; like `--f32-index` it only ever switches
the default off. `--float16` is still accepted and now a no-op.
Values beyond +-65504 are refused, so f32 remains the choice for
unnormalised vectors — the upgrade note says so.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-24 19:39:33 -05:00
osobhandClaude Opus 5.5 dbaf3f505d bench(longmemeval): --float16, and float16 measured on real embeddings 
`longmemeval_bench --float16` builds every per-question store with
MemoryConfig::float16, so the vector stage searches half-rounded
embeddings exactly as such a store holds them.

Full longmemeval_s (500 questions, ~494 turns each) with real
all-MiniLM-L6-v2 embeddings, f32 vs float16, on tank (CUDA): identical
at every Hit@k and MRR, turn and session level, in all eight modes —
bar RRF session MRR 0.9253 vs 0.9254 and one or two flips out of ~320
in which gold session ranks first. The f32 run reproduces the published
hybrid numbers exactly. The earlier float16 evidence was synthetic
clustered data only; this is the real-embedding check.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-24 19:21:14 -05:00
osobhandClaude Opus 5.5 c470244a6f feat(agent): HDF5Memory::search with source filters, re-ranking, confidence 
`HDF5Memory::search(query_embedding, query_text, &SearchOptions)` is the
store's full search path. `SearchOptions::new(k)` is plain hybrid search
with the tuned default fusion; each further stage is opt-in:

- `with_sources([..])`: only records from these source channels. The
  filter applies before ranking, so a filtered search still returns up
  to k results, normalised over what it can return. The HNSW pool is
  over-fetched in proportion to what the filter removes, and the allowed
  records are scanned exactly whenever that costs fewer distance
  evaluations than the index would (~pool x M) — and as the fallback if
  the pool comes back short. Keyword matches are filtered too.
- `with_rerank(ReRankConfig)` re-ranks a max(3k, 10) candidate pool by
  relevance, recency, source authority and activation;
  `with_confidence(ConfidenceConfig)` drops low-confidence results;
  `at_time(now)` pins the recency clock.

These were reachable only through the OpenClaw backend, which is now
`search` with both on. Its Hebbian boost now goes to the k results it
returns rather than the whole 3k candidate pool. `hybrid_search` and
`hybrid_search_with` are wrappers and unchanged (tested bit for bit).

Measured on tank (search_harness --options-study --full, 3 runs): at
100K every filter — 50%, 10%, 1% of the store, and records far from the
query — returns the exact filtered top 10, and none is slower than an
unfiltered search (1%: 2.3 ms vs 4.6 ms). Re-rank + confidence costs
about 3%. A first version decided between index and exact scan by pool
size vs store size; it measured 0.976 recall at 12.3 ms on the
far-from-query filter, which is why the rule compares costs instead.

Tests: tests/search_options.rs (filter correctness and full pages via
both paths, far-from-query fallback, edge cases, equality with
hybrid_search_with, re-rank recency, confidence, boost scope).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-24 16:35:42 -05:00
osobhandClaude Opus 5.5 d0db83812b feat(agent): MemoryConfig::float16 stores half-precision embeddings
CI / test-arm64 (pull_request) Successful in 1m19s
Details
CI / test (pull_request) Successful in 4m58s
Details 
The setting was persisted in /meta and otherwise ignored: embeddings
were always written as f32. It now does what it says.

clawhdf5-format:
- `DatasetBuilder::with_f16_data` writes IEEE binary16 (numpy float16),
  rounding to nearest-even, and `make_f16_type`.
- `clawhdf5_format::float16` holds the f32 <-> f16 conversions, the one
  implementation the writer, the reader and the agent all use. Checked
  against the `half` crate on 16.7M f32 values and round-trips all 65536
  half values; the h5py interop tests confirm the rounding matches
  numpy's bit for bit (4020 values incl. ties, subnormals, overflow).
- Reading little-endian float16 as f32 has a fast path.

clawhdf5-agent:
- A float16 store writes /memory/embeddings as half precision, and
  `MemoryCache::half_precision` rounds each embedding as it enters the
  cache (save, update, WAL replay, and on load of a store still f32 on
  disk), so memory and file agree bit for bit and a store searches the
  same before and after a reopen (tested).
- Values beyond +-65504 are refused with the new
  `MemoryError::InvalidEntry` rather than stored as infinity, on every
  save path; batches are all or nothing, and a rejected ephemeral entry
  stays in the ephemeral tier. Breaking for exhaustive matches.
- CLI: `create --float16`. Off by default.

Measured on tank, 384-dim, six runs alternating order, medians
(search_harness --float16-study --full): at 100K the file goes from
154.0 to 80.8 MiB (-48%), checkpoint 752 -> 512 ms, open 300 -> 252 ms;
vector recall@10 against an exact scan and hybrid_search latency do not
change. At 10K open is 3 ms slower. Also a test that h5py opens a whole
agent store, f32 and float16, and decodes every dataset.

Docs: README, BENCHMARKS.md ("float16 embedding storage"), CHANGELOG
(including the h5py interop fixes in the previous commit), CLAUDE.md.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-24 12:00:38 -05:00
osobhandClaude Opus 5.5 5e4aa1c6bf fix(format): files we write now open in h5py and libhdf5 
Two write-side bugs, both present in every release (the first at least
since v2.1.0), made libhdf5 refuse files written by clawhdf5. Our own
reader ignores both fields, and the interop suites only ever wrote f64
from our side, so nothing here caught them.

- Every f32 dataset: "sign bit position out of bounds". The float
  datatype encoder hard-coded the sign bit's position (bits 8-15 of the
  class bit field) to 63, which is right only for f64. It is now derived
  from the type: bit_offset + bit_precision - 1. This covered every
  agent store's embeddings, norms and activation weights.
- Every empty dataset: "invalid dataset size, likely file corruption".
  It was written with a real address and size 0, which trips libhdf5's
  `addr + size <= addr` overflow check. An empty contiguous dataset now
  gets the undefined address, as libhdf5 writes it. This covered every
  agent store without sessions or a knowledge graph.

Agent stores are rewritten in full at each checkpoint, so they become
readable at their next checkpoint on a fixed build; other files with f32
or empty datasets need rewriting. Both are recorded in
docs/known-issues.md.

Tests: the sign position byte for f32/f64, and h5py reading our f32
datasets (plain and chunked + deflate) bit for bit.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-24 12:00:26 -05:00
osobh 1cceb930b2 Merge pull request 'Feat/pure rust default and msrv' (#3) from feat/pure-rust-default-and-msrv into main
CI / test-arm64 (push) Successful in 1m20s
Details
CI / test (push) Successful in 5m59s
Details 
Reviewed-on: #3
 2026-09-23 16:29:33 +00:00
osobhandClaude Opus 5.5 fc7ae6549a build: declare Rust 1.92 as the MSRV and check it in CI
CI / test-arm64 (pull_request) Successful in 1m19s
Details
CI / test (pull_request) Successful in 5m23s
Details 
rust-version = "1.92" in [workspace.package], inherited by every crate.
1.92 is the floor: wgpu (clawhdf5-gpu) requires it, and the whole
workspace, Python bindings included, checks cleanly on it. ci-test.sh
reads the version from Cargo.toml and checks the workspace on exactly
that toolchain, so the manifests and the README badge cannot drift from
what actually builds. The badge said 1.75, below edition 2024's own
floor.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-23 11:05:26 -05:00
osobhandClaude Opus 5.5 735db117a7 build: pure-Rust zlib-rs as the default deflate backend 
The core crates (clawhdf5, -agent, -format, -io, -filters, -ann, -accel,
-netcdf4, -cli) now build no C by default: deflate defaults to zlib-rs,
a pure-Rust port of zlib-ng, and zlib-ng becomes the opt-in
`fast-deflate`, which overrides zlib-rs wherever it is enabled. A default
build no longer needs cmake or a C compiler.

Measured on tank, both builds run alternately, three rounds, medians:
zlib-rs is within 6% of zlib-ng on every HDF5 read and write (512x512
deflate-6 chunked write 1.458 vs 1.484 ms; 64 MB compressed read 64.4
vs 65.2 ms), and compressed output is byte-identical. Details in
BENCHMARKS.md, "Deflate backend".

Getting there took two fixes the first measurement exposed:

- zlib-rs needs `std` to detect SIMD at runtime. flate2 enables it via
  its default `runtime_detection`, which `default-features = false` had
  switched off, leaving zlib-rs 3.5x slower on inflate. The `zlib-rs`
  features now enable it.
- Both deflate paths streamed through flate2's 32 KiB read/write
  wrappers. They now hand the codec the whole chunk in one call, into a
  buffer sized up front (~5% on chunked writes). This also fixes a
  silent short read: the streaming reader returned a truncated stream's
  bytes without an error; a truncated chunk is now DecompressionError.
  In clawhdf5-filters, output longer than the stated size is now an
  error rather than silently cut off.

CI: ci-test.sh lints and tests the zlib-ng path, and fails if a
C-building crate (*-sys, cc, cmake) enters a core crate's default
dependency tree. The arm64 job no longer installs cmake.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-23 11:05:22 -05:00
osobhandClaude Opus 5.5 e9b37a9602 docs: bring the README up to date with the last five releases 
The README had fallen behind v2.3.0-v2.7.0, and parts of it were not
true. Checked every claim against the code and BENCHMARKS.md:

- Three of the six Quick Start snippets no longer compiled (Agent
  Memory, Consolidation, OpenClaw); all six now do.
- Hybrid search was described as RRF throughout. The default has been
  weighted 0.4/0.6 fusion since v2.5.0; re-ranking and confidence
  rejection run only in the OpenClaw backend.
- The `float16` feature does not halve embedding storage (the store
  always writes f32), `--features agent` enables nothing, "Source
  Isolation" is not wired in, and nothing backs "billion-scale" IVF-PQ.
- "Cryptographically verifiable" overstated an unkeyed, session-scoped
  FNV-1a ledger; "Zero C dependencies" was false while zlib-ng was the
  default deflate backend.
- Stale numbers: tests (1,650 -> 1,868), Rust badge (1.75 is below
  edition 2024's floor), 6.5 KB/record on disk (BENCHMARKS.md: 1.7 KB),
  consolidation and hybrid-search latency, and a feature-flag table
  broken by a paragraph pasted into it.
- The file schema, module table and crate map now match the code.

Adds a "What's new (v2.2 -> v2.7)" section for collaborators, leading
with the silent Extensible Array read bug fixed in v2.7.0. Footer links
point at git.redclaw.dev. CLAUDE.md: clawhdf5-migrate is the SQLite
migration tool, and MemoryConfig::compression is off by default.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
 2026-09-23 11:05:10 -05:00
osobhandClaude Opus 5 4bed8b3765 Merge docs/ci: record how CI is set up
CI / test-arm64 (push) Successful in 1m6s
Details
CI / test (push) Successful in 3m28s
Details 
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
 2026-09-22 04:48:17 -07:00
osobhandClaude Opus 5 36d689bc2c docs: record how CI is set up, and what broke it 
Two jobs, which runners serve them, and the two constraints that kept
the x86 job failing on every push until today: no JavaScript actions
(`rust:latest` has no `node`, and GitHub is not reachable from every
runner) and `cmake` for libz-ng-sys. Also notes that the Docker Hub
`latest` tag for the runner is frozen at 0.6.1, so it is not a way to
stay current.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
 2026-09-22 04:48:17 -07:00
osobhandClaude Opus 5 e7c08e06b4 Merge ci/fix-jobs: make both CI jobs actually run
CI / test-arm64 (push) Successful in 1m55s
Details
CI / test (push) Successful in 4m8s
Details 
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
 2026-09-21 20:37:51 -07:00
osobhandClaude Opus 5 c5049eb734 ci: stop depending on node, and install cmake where the build needs it 
Read from the job logs of the first run with the arm64 job, rather than
guessed:

- The x86 `test` job has been failing on every push, in about three
  seconds. It runs in `rust:latest` and starts with actions/checkout, a
  JavaScript action, and `rust:latest` has no `node`: exit 127 before a
  line of code was built. It is replaced with a plain git checkout (the
  image has git), and actions/cache, JavaScript for the same reason, is
  dropped.
- `test-arm64` got through checkout, toolchain, the aarch64 check and
  clippy, then failed building libz-ng-sys — pulled in by
  clawhdf5-format's default `fast-deflate` — because the host runner had
  no cmake. The x86 image lacks it too, so the x86 job would have hit the
  same wall one step later.

Both jobs now install cmake where they can (the Docker job and the x86
container run as root) and say plainly when they cannot (a host runner),
instead of failing inside a build script. vision-01 now has cmake.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
 2026-09-21 20:37:51 -07:00
osobhandClaude Opus 5 6f6bc97850 Merge ci/arm64: run the aarch64 kernels in CI
CI / test (push) Failing after 3s
Details
CI / test-arm64 (push) Failing after 37s
Details 
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
 2026-09-21 20:31:05 -07:00
osobhandClaude Opus 5 0cb72e8a60 ci: test the aarch64 kernels on an arm64 runner 
The NEON kernels in clawhdf5-accel — `dot_i8` including its SDOT path,
and the f32 NEON kernels that predate it — are cfg'd out on x86, so the
existing job has never compiled, linted or tested a line of them. They
were verified once, by hand, on a Raspberry Pi 5.

`test-arm64` runs on `linux_arm64`, which two runners serve in different
ways: vision-01 executes steps on the host with Rust preinstalled, and
vision-02 executes them in docker.gitea.com/runner-images. The job is
written to work in both: no `container:`, no JavaScript actions (those
are fetched from GitHub, which not every runner reliably reaches), and
an explicit `+stable` toolchain so a host's default — vision-01's is a
January nightly — is neither relied on nor changed. Fetches retry, since
one runner's outbound network was seen failing intermittently.

It lints the accel crate and tests accel, ann and format. It reports
rather than requires the dot-product extension: on a core without it the
plain-NEON kernel is the one that runs, and the tests cover whichever is
present.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
 2026-09-21 20:31:05 -07:00
osobhandClaude Opus 5 e338d58ad5 Merge feat/int8-default: new stores use the int8 index
CI / test (push) Failing after 2s
Details 
quantized_index defaults to true for new stores — smaller and faster at
equal recall on every configuration measured. Existing stores keep
their setting, and stores predating it stay f32, guarded by a real
v2.5.0 store committed as a test fixture.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
 2026-09-21 18:27:59 -07:00
osobhandClaude Opus 5 8b85d9364b feat(agent): new stores use the int8 vector index by default 
`MemoryConfig::quantized_index` now defaults to `true`. It holds a
quarter of the index memory and, with the exact re-score, is faster at
equal recall on every configuration measured: 1.63x the queries per
second on x86-64 (AVX2) and 1.18x on a Raspberry Pi 5 (NEON SDOT), with
builds 1.8x and 2.3x faster. The one argument for keeping it off — that
int8 search was slower on ARM — did not survive being measured.

Existing stores do not change. A store written with v2.6.0 or later
keeps its persisted setting. One written before the setting existed has
no stored value, and it loads as `false` rather than as the new default,
so reopening it never changes how its index is held. That case is
guarded by a real store written with the v2.5.0 CLI, committed as
`tests/fixtures/store_v2_5_0.h5` (6.8 KB): the test asserts it reopens
with an f32 index and still searches, and it fails if the load default
is changed to `true`.

The CLI needed more than a new default. `create --quantized-index`
assigned its value straight into the config, so under the new default
every CLI-created store would have been forced back to f32 unless the
caller knew to ask for int8. It is replaced by `--f32-index`, which only
ever switches the default off; `--quantized-index` is still accepted,
hidden, as a no-op, and the two conflict.

The whole agent suite passes under the new default, including the
brute-force recall oracle, now running on int8 plus re-score without
being asked to.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
 2026-09-21 18:27:59 -07:00
osobhandClaude Opus 5 6598a7d02f Merge feat/neon-int8: aarch64 int8 dot product, verified on a Pi 5
CI / test (push) Failing after 2s
Details 
SDOT and plain-NEON kernels for dot_i8, tested bit-exact against scalar
on real ARM. At equal recall the quantised index is 1.18x f32 on a Pi 5
and builds 2.3x faster. Also corrects an unmeasured claim that it was
slower than f32 on ARM.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
 2026-09-21 17:34:51 -07:00
osobhandClaude Opus 5 114a2dfcba docs: measured ARM numbers, and a correction 
On a Raspberry Pi 5 at N = 100 000 and equal recall (0.9940 vs 0.9945),
medians of three runs:

  f32             33 413 ms build   6 164 QPS
  int8 scalar     18 950 ms         ~6 190 QPS   (what v2.7.0 shipped)
  int8 NEON      ~17 000 ms          6 640 QPS
  int8 SDOT       14 464 ms          7 267 QPS   1.18x f32, 2.3x build

The docs said quantised search stayed off by default because aarch64
"falls back to the scalar loop, where the original trade still
applies" — that it was ~13% slower than f32 there, as on x86. That was
extrapolated rather than measured, and it was wrong: x86's portable
baseline is SSE2 against hand-written AVX2 f32 kernels, but on aarch64
NEON is the baseline and the scalar loop vectorises well, so it already
matched f32. Corrected in BENCHMARKS.md, README.md and CLAUDE.md; the
released v2.7.0 changelog entry is left as it was and the correction is
recorded in a new one.

Labelled as Pi 5 figures throughout — a Pi's memory bandwidth and cache
are far below an M-series or flagship phone, so the ratios will move.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
 2026-09-21 17:34:51 -07:00
osobhandClaude Opus 5 56a8c2f3d0 feat(accel): aarch64 int8 dot product — SDOT and plain NEON 
`dot_i8` had an AVX2 kernel and a scalar fallback, so on aarch64 the
quantised HNSW index ran the scalar loop. It now dispatches to one of
two NEON kernels:

- `dot_i8_dotprod`: the ARMv8.2 dot-product instruction, `SDOT`, which
  multiplies and accumulates sixteen i8 pairs into four i32 lanes per
  instruction. Present on Cortex-A76 and later (Raspberry Pi 5, current
  Android phones), Neoverse-N1 (Graviton2, Ampere Altra) and every Apple
  Silicon generation. Issued as inline assembly because the `vdotq_s32`
  intrinsic is still behind the unstable `stdarch_neon_dotprod` feature;
  inline asm is stable on aarch64.
- `dot_i8`: plain NEON for cores without the extension — `vmull_s8`
  widens to i16 (even -128 * -128 fits) and `vpadalq_s16` folds adjacent
  pairs into i32 accumulators, so nothing overflows.

Selected at runtime with `is_aarch64_feature_detected!("dotprod")`.

Verified on a Raspberry Pi 5 (Cortex-A76, `asimddp` present), not just
compiled — the aarch64 code is cfg'd out on x86, so x86 CI never builds
or lints it:

- both kernels bit-exact against scalar at every length, tails and
  extremes included. Each is tested directly rather than through
  dispatch, because dispatch only takes one path on a given CPU: on the
  Pi, testing through it alone would never have run the plain-NEON
  fallback at all.
- mutation-checked: dropping the SDOT kernel's second accumulator fails
  at length 32, and using the low half twice in the NEON kernel fails at
  length 16 — the first lengths that exercise each.
- the ANN suite passes, including int8 recall against ground truth.
- clippy clean with -D warnings on aarch64.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
 2026-09-21 17:34:51 -07:00
osobhandClaude Opus 5 7b16dc90d6 Merge release/v2.7.0
CI / test (push) Failing after 2s
Details 
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
 2026-09-20 18:24:07 -07:00
osobhandClaude Opus 5 4a5544da1d chore(release): v2.7.0 
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
 2026-09-20 18:24:07 -07:00
osobhandClaude Opus 5 f4c6d43a3f Merge feat/hnsw-tuning: HNSW parameters are configurable
CI / test (push) Failing after 1s
Details 
Graph degree and both candidate-list sizes were constants, so recall
could not be traded against memory or query speed. Now MemoryConfig
fields, persisted with the store, defaulting to today's behaviour.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
 2026-09-20 17:53:11 -07:00
osobhandClaude Opus 5 e5e087f9ab feat(agent): expose the HNSW parameters in MemoryConfig 
Graph degree and the build- and query-time candidate list sizes were
constants, so a deployment had no way to trade recall against memory or
query speed. They are now `MemoryConfig::hnsw_m`,
`hnsw_ef_construction` and `hnsw_ef_search`, persisted with the store
and defaulting to exactly the previous behaviour (16, 64, and a query
list that scales with `k`).

Two things the straightforward version would have got wrong:

`clawhdf5-ann` asserts a graph degree of at least 2, so a configured 0 —
from a file, or from a caller reading 0 as "use the default" — aborted
the process inside the index builder. The store clamps instead, and a
test covers it: removing the clamp makes that test panic rather than
fail.

`ef_search` and the candidate pool handed to score fusion were the same
number. Tying the pool to the new setting would mean lowering `ef` for
speed also narrows what fusion sees, quietly degrading hybrid results
through a knob that looks like it only costs time. They are now
independent.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
 2026-09-20 17:53:11 -07:00
osobhandClaude Opus 5 16c9ee0554 Merge perf/mmap-open: open a store without copying the whole file
CI / test (push) Failing after 1s
Details 
read_from_disk mapped the file and then copied the whole mapping for
File::from_bytes, which maps it itself. Store open is ~28% faster
(455 ms -> 327 ms at 100k x 384); peak memory is unchanged, because the
peak falls after the parse during the index build.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
 2026-09-20 17:49:54 -07:00
osobhandClaude Opus 5 1d767e3b93 perf(agent): open a store without copying the whole file 
`read_from_disk` memory-mapped the file and then copied the entire
mapping into a `Vec` to hand to `File::from_bytes` — but `File::open`
memory-maps it itself whenever the facade's `mmap` feature is on, which
it is by default. So every open mapped the file, memcpy'd all of it, and
parsed the copy.

Store open at 100k x 384: 455 ms -> 327 ms, about 28% faster (two runs
after the change, 326.8 and 328.1 ms).

Peak memory is unchanged, which is worth saying because the opposite is
the natural assumption. The footprint harness now tracks a high-water
mark next to the retained figure, and it shows the peak falling after
the parse, during the index build — so a buffer allocated and freed
inside the parse never reaches it. Confirmed rather than assumed:
holding a deliberate extra copy of the whole file across the parse
leaves the peak exactly where it was, which is also what proved the
instrument was working before trusting its answer.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
 2026-09-20 17:49:47 -07:00
osobhandClaude Opus 5 a91df3f1c3 Merge fix/array-checksums: chunk index checksums are verified
CI / test (push) Failing after 2s
Details 
Fixed and Extensible Array structures all carry a Jenkins checksum that
was ignored. A single flipped bit in a chunk address parses cleanly and
points inside the file, so without the check the reader returns another
chunk's bytes as data.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
 2026-09-20 17:41:48 -07:00
osobhandClaude Opus 5 b41272487a fix(format): verify Fixed and Extensible Array checksums 
Every structure in both chunk indexes — header, index block, super
block, data block and each data block page — carries a Jenkins lookup3
checksum, and all of them were parsed past and ignored.

What that costs is not a warning but correct data. Flip one low bit of a
chunk address and the index still has the right shape, the address still
lands inside the file, and the reader returns whatever bytes now sit
there as that chunk's contents. Nothing else in the parse can tell.

Verified in both directions. The checksums accept files written by
HDF5 2.0 from 100 to 200 000 chunks — dense, sparse, gzip-filtered and
paged — which also confirms the block layouts byte for byte, since a
wrong offset would fail every file. And an interop test corrupts an
address to check the read fails instead of returning data: removing the
verification makes that test fail with "corruption produced data instead
of an error", which is what it is there to prove.

The first version of that test passed with verification disabled — it
corrupted a byte a structural check already rejected, so it proved
nothing. Worth recording, since a test that passes for the wrong reason
looks exactly like coverage.

Hand-built fixtures now stamp real checksums, as HDF5 writers do, and
the Extensible Array ones no longer describe the superseded layout.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
 2026-09-20 17:41:41 -07:00
osobhandClaude Opus 5 0bc7a293ae Merge feat/int8-kernel: int8 index is faster than f32 on AVX2
CI / test (push) Failing after 3s
Details 
The quantised HNSW index was measured against f32 with a scalar loop on
one side and clawhdf5-accel's AVX2 kernels on the other, so the ~13%
throughput cost attributed to quantisation was a missing kernel.

With clawhdf5_accel::dot_i8 in place, at equal recall the quantised
index answers 1.63x the queries per second and builds 1.8x faster,
holding a quarter of the vectors. Still off by default, now because the
kernel is AVX2-only and aarch64 falls back to scalar.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
 2026-09-20 17:32:40 -07:00
osobhandClaude Opus 5 dea02f5214 docs: the int8 index is faster than f32 on AVX2, not slower 
Measured with `clawhdf5_accel::dot_i8` in place, medians of three
alternating runs at N = 100 000 x 384, same binary:

  build      f32 3197 ms   int8 1778 ms   int8+re-score 1826 ms
  ef = 64    f32 13 399 QPS @ 0.9945   int8+re-score 21 848 QPS @ 0.9940

So at equal recall the quantised index is 1.63x the queries per second
and 1.8x the build speed, holding a quarter of the vectors. The earlier
"~13% of QPS" figure compared a scalar int8 loop against hand-written
AVX2 f32 kernels and was measuring the missing kernel; it is kept in
BENCHMARKS.md with that explanation rather than quietly replaced.

Still off by default, now for portability rather than performance: the
kernel is AVX2-only and aarch64 falls back to scalar, where the original
trade applies. A NEON kernel would settle it.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
 2026-09-20 17:30:56 -07:00
osobhandClaude Opus 5 97e65f2adf feat(accel): runtime-dispatched int8 dot product 
The int8-quantised HNSW index compared vectors with a scalar loop that
the compiler vectorised for the x86-64 baseline (SSE2), while the f32
path it was measured against goes through `clawhdf5-accel` and runs
AVX2. So the ~13% throughput cost recorded for `quantized_index` was a
missing kernel rather than a property of int8.

`clawhdf5_accel::dot_i8` adds a scalar fallback and an AVX2 path:
sign-extend each 16-byte half to i16, then `madd_epi16`, which
multiplies and sums adjacent pairs straight into i32 lanes. It is
dispatched through the same detected backend as the f32 kernels, and
the index now calls it.

Integer arithmetic, so the SIMD path must agree with scalar bit for
bit — tested at lengths that are and are not multiples of the block,
and at the -128 extreme for overflow.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
 2026-09-20 17:30:56 -07:00
osobhandClaude Opus 5 fb58300b3f Merge fix/audit-robustness: Extensible Array corruption, B-tree v2 crash
CI / test (push) Failing after 3s
Details 
Two read-path bugs found by auditing the improvement plan against the
code, both reaching every release up to v2.6.0.

Datasets indexed by an Extensible Array — any dataset with one
unlimited dimension — returned data from the wrong chunks past their
first few dozen, silently. The only fixture in the suite had three
chunks, inside the inline limit, so no test had ever read a data block.

A crafted file could abort the process through unbounded B-tree v2
recursion, or exhaust memory through shared subtrees. Both now refuse
in under a millisecond, and the fuzz target that should have caught it
(it only fuzzed header parsing) now walks the tree.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
 2026-09-20 17:24:55 -07:00
osobhandClaude Opus 5 eb196e824f test: cover Fixed Array chunk indexes against real files 
The Extensible Array bug reached a release because no fixture had more
chunks than fit inline, so its data blocks were never read. The Fixed
Array index had the same blind spot: nothing exercised it above a
handful of chunks, and nothing reached the paged layout at all.

Checked at 100, 5 000 and 200 000 chunks plus a sparse dataset that
leaves whole pages uninitialised. It is correct throughout — it does
keep its page-init bitmap inside the data block, which is the
difference from the Extensible Array that made assuming otherwise a
bug. Adding the tests so that stays true.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
 2026-09-20 17:20:12 -07:00
osobhandClaude Opus 5 367faad7f7 fix(format): read Extensible Array chunk indexes correctly 
A dataset with exactly one unlimited dimension — the ordinary
append-only case — is indexed by an Extensible Array. Only its first
few chunk entries (4 by default) sit inline in the index block, and
everything past them was read with the wrong layout. In the default
shape the 37th chunk onward came back from the wrong place: a
400-chunk dataset returned 364 wrong values while reporting success,
and beyond about a thousand chunks the read failed outright. Silently
wrong data is the worse half of that.

It survived because the only Extensible Array fixture in the suite had
three chunks — inside the inline limit — so no test ever reached a data
block.

Four layout errors, each confirmed against files written by HDF5 2.0 and
against the library source rather than inferred:

- super block `u` owns 2^(u/2) data blocks, not 2^u;
- each holds 2^((u+1)/2) * data_blk_min_elmts elements — the two
  quantities double every *other* level, a half step apart;
- a super block carries a block-offset field before its data block
  addresses, which was not skipped;
- the page-init bitmap belongs to the super block, one bit per page
  packed across all of its data blocks and read MSB-first, rather than
  living inside the data block; a paged data block also ends its prefix
  with a checksum before the first page.

Where the spec left room for doubt the file settled it: decoding a
paged block's elements and reading the chunk values they address
identifies the mapping exactly, and the bitmap's 68 set bits matched
the 34 data blocks x 2 pages that 200 000 elements need, which only
holds MSB-first.

New interop tests cross every boundary — 4, 37, 400, 5 000 and 200 000
chunks, the last with paged data blocks — plus sparse (uninitialised
pages taking fill values), gzip-filtered elements and a 2-D dataset.
All three fail against the old traversal.

Writing is untouched; this was a read-path bug.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
 2026-09-20 17:18:59 -07:00
osobhandClaude Opus 5 0901fb1499 test(fuzz): fuzz B-tree v2 traversal, not just header parsing 
The existing target only called `BTreeV2Header::parse`, so the
recursive walk behind it — where a node that is its own child overflowed
the stack — was never fuzzed at all. Parsing also requires a valid
Jenkins checksum, which random input essentially never produces, so
almost every input stopped at the first branch.

The target now walks the tree after a successful parse, and also builds
a header straight from the input bytes so the traversal is reachable
without forging a checksum.

Checked both ways: against the unfixed traversal libFuzzer finds the
stack overflow (ASan: stack-overflow), and against the fix that same
input executes in 0 ms and 34.7 million further runs produce no crash,
timeout or OOM.

Corpora and crash artifacts stay out of the repository; the two crafted
inputs are covered by unit tests instead.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
 2026-09-20 17:03:21 -07:00
osobhandClaude Opus 5 e9aeb110b7 fix(format): bound B-tree v2 traversal against crafted files 
Traversal recursed one frame per level with the depth taken from the
file (a u16), and followed child addresses without asking whether they
were shared. Two crafted inputs, both reproduced before fixing:

- A node listing itself as its own child, under a header claiming 65 535
  levels, overflowed the stack and aborted the process — SIGABRT, not an
  error a caller can handle — from under 100 bytes.
- Levels whose children all point at one shared node below reached it
  fan-out^depth times: 29.5 million records in 8 s from ~5 KB, and one
  more level would exhaust memory.

Depth is now capped at 64, as the fractal heap already was; no real tree
approaches it, since even at the minimum fan-out of two that is over
2^64 records. And traversal stops once it has produced more records
than the file has bytes to hold them — a valid tree stores each record
once in its own bytes, so this bounds shared subtrees without trusting
the header's own `total_records`. Both inputs now fail in under a
millisecond.

Every B-tree v2 user goes through this collector: dense attributes, v2
groups, shared messages and chunk indexes. To show the budget never
refuses a real file, a new interop test has HDF5 2.0 write a depth-2
chunk index with 40 000 records and reads back all 160 000 values; it
fails when the budget is deliberately made too tight.

Also corrects `BM25Index::search`, which claimed to use Block-Max WAND.
It scores exhaustively, and pruning would not help the store:
`hybrid_search` needs every score because fusion normalises over them.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
 2026-09-20 16:56:05 -07:00
osobhandClaude Opus 5 5889b378e9 Merge release/v2.6.0
CI / test (push) Failing after 1s
Details 
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
 2026-09-20 06:01:38 -07:00
osobhandClaude Opus 5 18dc35f7e5 chore(release): v2.6.0 
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
 2026-09-20 06:01:31 -07:00
osobhandClaude Opus 5 e17ab0ceef ci: name the interop interpreter instead of relying on $GITHUB_PATH
CI / test (push) Failing after 1s
Details 
The workflow already installed h5py into /opt/interop and set
CLAWHDF5_REQUIRE_INTEROP=1, but it reached the tests only by appending
that venv to $GITHUB_PATH, which Gitea's runner does not reliably
propagate into test subprocesses. If `python3` resolved to the system
interpreter instead, every interop suite would skip. Setting
CLAWHDF5_PYTHON outright removes the question: together with
REQUIRE_INTEROP the suites either run or the build goes red.

Verified both directions locally — with a venv the four suites run 94
tests green; with a bogus interpreter and REQUIRE_INTEROP=1 the facade
and netCDF4 suites fail 22 tests rather than skipping.

Also documents creating the local `.venv` that `ci-test.sh` detects.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
 2026-09-20 04:55:09 -07:00
osobhandClaude Opus 5 105cf13347 docs: record the silent interop skip in known-issues
CI / test (push) Failing after 3s
Details 
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
 2026-09-19 20:45:21 -07:00
osobhandClaude Opus 5 0529f72a2c Merge feat/ann-quantisation: optional int8 vector index; interop suites run again 
Cuts a loaded store's memory from 2.72x to 1.74x the raw vectors at 100k
x 384 via MemoryConfig::quantized_index, with recall held at the f32
index's level by re-scoring candidates against the exact embeddings the
store already holds. Off by default: it trades ~13% of QPS for the
memory.

Also restores the Python interop suites, which had been skipping
silently on this machine because no interpreter has h5py and PEP 668
blocks installing it into the system one.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
 2026-09-19 20:44:59 -07:00
osobhandClaude Opus 5 a29c1b224b test: let the interop suites find a Python that actually has h5py 
Every Python interop suite had stopped running on this machine: the h5py
writer round-trips, the facade suite, netCDF4 and the reference files.
`python3` is 3.14, nothing on the box has h5py, and PEP 668 refuses to
install it into a system interpreter at all — so the availability probes
all returned false and each suite skipped without failing.

A silent skip here is exactly how the v5 compound-datatype bug reached a
release, so the probes now read `CLAWHDF5_PYTHON` and `ci-test.sh` picks
up `.venv/bin/python` on its own. The detection sits at the top of the
script rather than beside the interop step, because the non-ignored
suites run in the earlier `cargo test` step and would otherwise still
miss it. `CLAWHDF5_REQUIRE_INTEROP=1` continues to turn a skip into a
failure.

Verified against a venv with h5py 3.16 / HDF5 2.0.0: 94 interop tests
across the four suites, all passing.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
 2026-09-19 20:44:54 -07:00
osobhandClaude Opus 5 c0a9206703 feat(agent): optional int8 vector index, re-scored against exact embeddings 
`MemoryConfig::quantized_index` stores the HNSW index's own copy of the
embeddings as i8 rather than f32. At 100k x 384 that takes the index from
266 to 123 MiB and the whole reopened store from 399 to 256 MiB — 2.72x
to 1.74x the raw vectors, the largest remaining item in the footprint.

Quantised distances are approximate and `ef` cannot compensate, because
the loss is in the distances rather than in the graph: recall@10 tops out
at 0.967 against f32's 0.9995 and does not move between ef=128 and
ef=256. The store already holds the exact embeddings, though, so when the
index is quantised the query path re-scores the candidate pool against
them before fusion. That restores recall (0.9940 vs 0.9945 at ef=64) and
costs about 13% of QPS.

Off by default: it trades query speed for memory and which side is worth
more depends on the deployment. The flag is persisted in `/meta`, so a
reopened store does not silently revert to four times the index memory,
and the sidecar graph is rehydrated into the configured storage.

Also on the CLI as `create --quantized-index`.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
 2026-09-19 20:40:37 -07:00
osobhandClaude Opus 5 57756e69ec feat(ann): optional int8 storage for the index's vector copy 
The HNSW index keeps its own copy of every vector, which at 100K x 384
f32 is ~146 MiB — the largest single item in the 2.43x footprint now
that the agent stores embeddings once. `Storage::Int8` cuts that copy to
a quarter by scaling each row to i8.

The scale is per row, not global. Unit-length rows in d dimensions have
components around 1/sqrt(d), so a fixed [-1, 1] scale spends fewer than
12 of the 255 levels on a 128-dimensional vector; measured against an
exact ranking that gives 0.35 top-10 overlap. Scaling each row by its
own largest component uses the full range and brings it to 0.99.

Quantised distances still cost recall on their own, and `ef` does not
buy it back because the loss is in the distances rather than the graph:
at N=100K recall@10 tops out at 0.967 against f32's 0.9995. Re-scoring a
wider candidate pool against the exact vectors removes the gap
(0.9940 vs 0.9945 at ef=64) for ~13% of query throughput and ~16% of
build time. That is the intended use, so it is what the test asserts —
against ground truth, not against the f32 index, whose own mistakes a
re-scored search is entitled to get right.

Default is unchanged: `Storage::Float32`, chosen by every existing
constructor. Serialized indexes carry f32 vectors and no storage tag, so
a quantised index is rebuilt rather than loaded; `compact()` keeps the
storage it was given.

The harness grows `--int8` and `--rerank` axes, and reports the storage
in each table header.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
 2026-09-19 20:35:24 -07:00
osobhandClaude Opus 5 6ad8ceb426 Merge feat/vector-footprint: store embeddings once; footprint measurement
CI / test (push) Failing after 1s
Details 
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
 2026-09-19 20:19:43 -07:00
osobhandClaude Opus 5 2e7e0456c1 perf(agent): store embeddings once, not twice 
MemoryCache held every embedding in two places: a `Vec<Vec<f32>>` and a
flattened copy for the batched kernels, kept in lock-step on every push,
update and compaction. A store loaded from disk therefore carried the corpus
twice, plus one heap allocation per entry.

A new `cache::Embeddings` owns just the flat `[N x dim]` buffer and indexes
into it, so `embeddings[i]` still reads as a `&[f32]` row. The batch kernels
take a `VectorSet` (implemented for both `Embeddings` and `Vec<Vec<f32>>`)
instead of `&[Vec<f32>]`, so their callers and tests are unchanged. Loading no
longer unflattens what it just read.

100k 384-dim entries, reopened from disk: 505 -> 357 MiB, 3.44x -> 2.43x the
raw vectors. Recall (1.0000 at ef=64) and query latency are unchanged.

Rows are now always exactly `dim` long, shorter ones zero-padded. The old
representation allowed ragged rows, which silently misaligned the flattened
copy — every row after a wrong-length embedding — and `update` carried a
comment about falling back to a rebuild to avoid exactly that. It is now
unrepresentable. A record saved without an embedding holds a zero row and is
told apart by its norm, which is what `total_embeddings` now counts.

Measured with a counting allocator rather than RSS: freeing a structure
returns its pages to the allocator's pool, not the OS, so an RSS reading from
inside the process showed the two representations as identical.

Breaking: MemoryCache::embeddings changes type, embeddings_flat is replaced by
flat_embeddings(), rebuild_flat() is a deprecated no-op.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
 2026-09-19 20:17:55 -07:00
osobhandClaude Opus 5 dc0113d015 Merge feat/temporal-reranking: re-ranking keeps the retrieval score; recency metric
CI / test (push) Failing after 2s
Details 
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
 2026-09-19 20:05:52 -07:00
osobhandClaude Opus 5 8ea455bbcb fix(agent): re-ranking threw away the retrieval score 
reranker::rerank built its combined score from temporal decay, source
authority and Hebbian activation. RerankInput carried no relevance score, so
it could not have used one: re-ranking a candidate pool reordered it purely by
age and discarded the retriever's ordering. The OpenClaw backend re-ranks
every search, so that was its shipping behaviour.

Measured over the full LongMemEval haystack (500 questions, real MiniLM
embeddings), ordering by metadata alone costs 40.6pp of Hit@1 (11.0% vs 51.6%)
and two thirds of MRR (0.1829 vs 0.6430) — the results are the newest memories
in the pool rather than the ones answering the question.

RerankInput::relevance and ReRankConfig::relevance_weight (1.0 by default)
make relevance lead, with the metadata signals breaking near-ties. Retrieval
is preserved (Hit@1 52.0%, +0.4pp against no re-ranking; MRR -0.003) and
recency discrimination improves 6-7pp, from chance to ~52%.

A half-life sweep (1, 7, 30, 90 days) moves recency 1.4pp and MRR 0.003 —
inside the noise — because the temporal term is capped by its weight while
relevance gaps are larger. The 24-hour default is kept: there is no measured
reason to change it. The two ends of the trade-off are recorded in
BENCHMARKS.md rather than just the good news.

Breaking: RerankInput and ReRankConfig gained fields.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
 2026-09-19 20:03:48 -07:00
osobhandClaude Opus 5 1e18ff5a86 style(bench): gate the rerank-sweep flag and helper on the embeddings feature 
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
 2026-09-19 19:42:05 -07:00
osobhandClaude Opus 5 306a35347c bench: use real session dates, and measure recency discrimination 
Two gaps in the LongMemEval harness, both of which had to close before any
recency feature could be judged.

The store was fed a synthetic counter (ts += 1.0 per turn) and the dataset's
own `haystack_dates` were ignored. Session order happened to be chronological,
so ordering was right, but the intervals were fiction — and exponential decay
is a function of the interval, so anything time-aware was being measured
against made-up ages. Dates are now parsed (civil-from-days, pinned against
reference values) and turns are spread over the minutes after their session
start; an unparseable date falls back to position so order still holds.

`newest_gold_first` measures what recall cannot. On a `knowledge-update`
question LongMemEval labels *both* the stale session and the one that
supersedes it as gold, so returning either scores as a hit even though only
one answers the question. The new metric asks whether the newest gold session
outranked the older ones. The current retriever scores 43-45% on it across
every mode — chance — which is the gap a temporal signal is supposed to close.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
 2026-09-19 19:41:44 -07:00
osobhandClaude Opus 5 4c60398b30 Merge release/v2.5.0
CI / test (push) Failing after 3s
Details 
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
 2026-09-19 18:25:26 -07:00
osobhandClaude Opus 5 7155409202 chore(release): v2.5.0 
Bump all workspace crates, the node package and pyproject to 2.5.0, fold the
two unreleased sections together and add upgrade notes for the behaviour
changes.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
 2026-09-19 18:22:56 -07:00
osobhandClaude Opus 5 64d9c5f171 Merge feat/bm25-tokenizer: optional keyword stemming, measured and left off by default
CI / test (push) Failing after 2s
Details 
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
 2026-09-19 18:22:27 -07:00
osobhandClaude Opus 5 84a39ef3c5 feat(agent): optional keyword stemming, measured and left off by default 
The keyword stage had no stemming, so "training" and "trains" were unrelated
terms. bm25::TokenFilter::Stemmed strips common English inflections (plurals,
-ing/-ed, with consonant un-doubling) from documents and queries alike;
BM25Index::build_with and HDF5Memory::set_token_filter select it, and the
index records which filter built it so a stale one is rebuilt rather than
mixed.

Measured over the full LongMemEval haystack (500 questions, real MiniLM
embeddings) rather than adopted on principle — and it is a trade, not a win:

  BM25 only         Hit@1 53.8%  Hit@5 75.0%  Hit@10 81.6%  MRR 0.6320
  BM25 stemmed      Hit@1 52.0%  Hit@5 77.8%  Hit@10 84.0%  MRR 0.6320
  Hybrid 0.4/0.6    Hit@1 51.6%  Hit@5 81.4%  Hit@10 87.8%  MRR 0.6430
  Hybrid stemmed    Hit@1 50.2%  Hit@5 81.4%  Hit@10 88.2%  MRR 0.6394

Conflation buys depth and costs the top rank: on BM25 alone MRR is unchanged
to four decimal places, the deeper gains exactly offsetting the rank-1 loss.
On the shipping hybrid configuration the vector stage already supplies most of
that recall, so the trade is narrower and slightly negative. Default stays
Plain; Stemmed is there for callers who want Hit@5/@10 over rank-1 precision.

The stemmer is deliberately conservative — it only strips inflections, and
only when the stem stays long enough to be meaningful, since an aggressive one
also conflates unrelated words. Tests pin both the pairs that must meet and
the pairs that must not.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
 2026-09-19 18:20:19 -07:00
osobhandClaude Opus 5 55ed87d2e8 Merge feat/retrieval-quality: tuned fusion defaults, RRF measured, query-expansion fixes
CI / test (push) Failing after 2s
Details 
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
 2026-09-19 17:40:46 -07:00
osobhandClaude Opus 5 6531158d9f fix(agent): query expansion panicked on non-ASCII and rewrote text inside words 
Probing what QueryExpander::expand actually produces for LongMemEval questions
turned up two defects in the same helper.

`replace_word_case_insensitive` did a plain substring replace despite its
name, so acronym expansion fired inside ordinary words: "training" became
"trArtificial Intelligencening" ("ai") and "programming" became
"Pull Requestogramming" ("pr"). Nearly every acronym expansion of prose was
corrupt. Matching now requires word boundaries at both ends; real acronyms
(API, database) still expand in both directions.

The same helper searched `text.to_lowercase()` and then sliced `text` with the
offsets it found. That holds only while lowercasing preserves byte length, and
it does not — Turkish 'İ' is 2 bytes and lowercases to 3. Offsets after such a
character drifted, so output was silently corrupted ("İstanbul AI trip" lost a
character) or the slice landed inside a character or past the end and
panicked: `expand("İ AI")` was enough, from a plain query string. Matching now
walks the original string, comparing case-insensitively char by char, so
offsets are always valid.

Regression tests cover both, plus whole-word matching at string edges. The
morphological rules remain crude ("during" -> "dured"); that is a quality
limit, not a correctness bug, and is now documented as a reason to measure
before enabling expansion on a retrieval path.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
 2026-09-19 17:25:20 -07:00
osobhandClaude Opus 5 aa92fef7bb bench: measure RRF against the weighted sum — weighted wins, RRF not adopted 
Reciprocal rank fusion was implemented but reachable only as a free function
over a linear scan, so its merits had never been tested. With both running
over the same HNSW + BM25 candidates on the full LongMemEval haystack (500
questions, real MiniLM embeddings):

  weighted 0.4/0.6   turn Hit@1 51.6%  Hit@5 81.4%  MRR 0.6430
  RRF k=60           turn Hit@1 45.0%  Hit@5 78.8%  MRR 0.5967

RRF lands almost exactly where the old 0.7/0.3 weighting did, and for the same
reason: it combines the stages by rank with equal influence, but on this corpus
BM25 alone beats the vector stage by 17.8pp at Hit@1, so treating them as peers
costs rank-1 accuracy. RRF's advantage is robustness when the stages' scores
are not comparable and there is nothing to tune against; here there is, so the
weighted sum stays the default. Recorded in BENCHMARKS.md with the reasoning,
including that this is a property of the corpus rather than a defect in RRF.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
 2026-09-19 17:18:09 -07:00
osobhandClaude Opus 5 29baabbed2 feat(agent): selectable fusion; adopt the measured 0.4/0.6 default weights 
BENCHMARKS.md has recorded since the weight sweep that the 0.7/0.3 default is
strictly dominated by 0.4/0.6 over the full LongMemEval haystack, but the
shipping code never adopted it: unified_search and the OpenClaw backend both
passed 0.7/0.3. Re-running the sweep here (500 questions, real MiniLM
embeddings on a GPU) reproduces it — turn-level Hit@1 51.6% vs 44.2%, Hit@5
81.4% vs 79.2%, Hit@10 87.8% vs 85.8%, MRR 0.6430 vs 0.5856 — so both now use
hybrid::DEFAULT_FUSION, which is that operating point and carries the
reasoning. A unit test pins it.

Fusion is also selectable now. hybrid::Fusion is either Weighted { vector,
keyword } or Rrf { k }; hybrid::fuse applies either to one candidate list per
stage, and merge_vector_keyword / hybrid_search delegate to it, so the public
API is unchanged. New HDF5Memory::hybrid_search_with and
hybrid::hybrid_search_fused take a Fusion. Reciprocal rank fusion was
implemented but reachable only as a free function over a linear scan, so it
had never been compared with the weighted sum on equal terms; it is now a mode
in the LongMemEval bench (measurement to follow).

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
 2026-09-19 16:57:51 -07:00
osobhandClaude Fable 5.1 b23946e62d Merge feat/hdf5-read-path: partial reads, B-tree v2 chunk index, faster full reads, auto-chunking, H5T_STD_REF
CI / test (push) Failing after 2s
Details 
Co-Authored-By: Claude Fable 5.1 <[email protected]>
 2026-09-19 16:17:26 -07:00
osobhandClaude Fable 5.1 52cfcf20b2 feat(format): parse H5T_STD_REF references and decode object references 
HDF5 1.12 revised the reference datatype (class 7) in datatype message version
4: reference types 2-4 are the new H5T_STD_REF object / dataset-region /
attribute references. Datatype::parse rejected them with
InvalidReferenceType, so any dataset of that type was unreadable.

h5py cannot write this type, which is why it had never been tested. A real
file was produced by calling the libhdf5 bundled in the h5py wheel through
ctypes (H5T_STD_REF_g, H5Rcreate_object, H5Dwrite); the 2 KB result is
committed as tests/fixtures/std_ref_hdf5_2_0.h5 with its generator,
gen_std_ref.py.

- ReferenceType gains Object2, DatasetRegion2 and Attribute, accepted only
  from datatype version 4.
- read_object_references decodes Object2 elements: type(1) flags(1)
  token_size(1) token, zero-padded to the element size; the token is the
  target's object header address. A null reference decodes to the undefined
  address; an external reference, a wrong type byte or a token that doesn't
  fit is an error.

The fixture test follows both references and checks they resolve to the
objects they were created from.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
 2026-09-19 14:24:04 -07:00
osobhandClaude Fable 5.1 05c665a898 feat(format): choose chunk dimensions automatically for large datasets 
Requesting a filter without chunk dimensions made the whole dataset a single
chunk. Any read, even one row, then decompresses everything, and a large
dataset cannot be decoded in parallel — which also made the new partial reads
pointless for such files.

auto_chunk_dims keeps datasets up to 1 MiB as one chunk (unchanged behaviour)
and splits larger ones by halving the dimensions in turn, so chunks keep
roughly the dataset's proportions, until a chunk is at most 1 MiB — h5py's
approach. An empty (unlimited, unwritten) dimension is treated as 1024. The
writer passes the element size through resolve_chunk_dims_for; the old
resolve_chunk_dims assumes 8-byte elements. Explicit with_chunks always wins.

Interop test: h5py reads an auto-chunked 13 MB deflate dataset, sees chunks
between 128 KiB and 1 MiB, and a small dataset still has one chunk.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
 2026-09-19 14:21:04 -07:00
osobhandClaude Fable 5.1 b36c6ec2af style(format): as_chunks_mut in the un-shuffle interleave (clippy) 
Co-Authored-By: Claude Fable 5.1 <[email protected]>
 2026-09-19 14:06:36 -07:00
osobhandClaude Fable 5.1 f3d63dbdcd perf(format): un-shuffle by interleaving fixed-width byte planes 
shuffle_decompress — on the read path of every compressed dataset, since
shuffle is applied automatically before compression — was the naive
`result[i * es + j] = data[j * n + i]`: a multiply and two bounds checks per
byte. It now interleaves fixed-width arrays of byte planes for element sizes
2/4/8/16 (bounds checks hoisted, vectorisable), with a chunked generic
fallback. The write-side shuffle was already optimised; this was the asymmetry
the survey flagged. Modest wall-clock effect now that decode is parallel
(chunked+deflate full read ~70 -> ~66 ms). Round-trip test over element sizes
1-24 and several lengths.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
 2026-09-19 14:06:10 -07:00
osobhandClaude Fable 5.1 0addf328bc perf(format): parallel cached decode and fewer copies on full reads 
Same-moment A/B on a 64 MB f64 dataset: chunked+deflate 110 -> 69 ms, chunked
72 -> 60 ms, contiguous 56 -> 30 ms.

- read_chunked_data_cached — the path the facade uses — decompressed chunks
  one at a time; only the uncached reader was parallel. Cache misses are now
  decoded in bounded batches (128), in parallel with the `parallel` feature.
- Every chunk was pushed into the 16 MiB chunk cache, which a larger dataset
  just churns (insert, evict moments later). Chunks are cached only when the
  whole dataset fits (new ChunkCache::max_bytes).
- Unfiltered chunks went file -> Vec -> aligned cache buffer -> output. They
  are copied straight from the file bytes.
- The facade's typed reads convert a contiguous dataset straight from the
  borrowed file bytes instead of copying it into a Vec first.
- The native little-endian fast paths allocated vec![0; n] and then overwrote
  it; they now fill an uninitialised buffer in one copy (native_le_to_vec).
  alloc_output requests zeroed memory from the allocator instead of reserving
  and filling.

The unit test that expected unfiltered chunks to land in the decompressed
cache now asserts the new design (index reused, cache not involved).

Co-Authored-By: Claude Fable 5.1 <[email protected]>
 2026-09-19 14:05:15 -07:00
osobhandClaude Fable 5.1 d668e45ab5 feat(format): read chunked datasets indexed by a version-2 B-tree 
With libver='latest', a chunked dataset with two or more unlimited dimensions
indexes its chunks with a v2 B-tree (layout v4, index type 5). Reading one
failed with "unsupported chunked layout version=4, index_type=Some(5)".

read_btree_v2_chunks decodes record types 10 (address + scaled offsets) and 11
(address, stored size, filter mask, scaled offsets). The width of the
stored-size field is taken from the record size the tree header declares
rather than re-deriving the library's formula. Scaled offsets are multiplied
back by the chunk dimensions with overflow checks.

The chunk-index dispatch existed four times (uncached, cached, sweep and
indexed readers). The three copies outside list_chunks now call it, so every
read path — and fill-value handling and partial reads — supports every index
type from one place.

h5py interop test: plain, gzip+shuffle, a 2500-chunk tree with internal nodes,
a sparse dataset with a fill value, and a strided hyperslab.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
 2026-09-19 13:59:18 -07:00
osobhandClaude Fable 5.1 c6a7bbfc67 perf(format): partial selection reads; out-of-range selections are errors 
read_raw_data_selection computed which chunks a selection intersects, threw
the answer away, decoded the entire dataset and picked elements out of it —
for contiguous layouts too. A 64x64 window of a 64 MB deflate dataset cost
105 ms, about half a full read; every selection cost the same whatever its
size.

New partial_read module: materialise only the selection's bounding box — the
overlapping rows of a contiguous dataset (straight from the file bytes) or the
overlapping chunks (only those are decompressed) — then run the existing
extractor over that buffer with the selection translated to the box origin, so
extraction semantics are exactly the full-read ones. It declines (falling back
to the old path) for All/None, compact/virtual/storage-less layouts, and boxes
covering more than half the dataset. That window now takes 0.39 ms, one row
2.7 ms, one column 5.2 ms.

Selections are validated against the dataset shape first. They were not: a
hyperslab past an edge came back padded with zeros and a point with an
out-of-range column wrapped into the next row, returning the wrong element
with no error. Now FormatError::SelectionOutOfBounds (also rank mismatch and
overlapping blocks); the facade's fill-aware path validates too.

Tests: equivalence against a reference extraction from a full read over 60
random hyperslabs/point lists per layout (contiguous, chunked, deflate) for
ranks 1-3. New read_harness bench binary with before/after in BENCHMARKS.md.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
 2026-09-19 13:57:14 -07:00
osobhandClaude Fable 5.1 3027380979 Merge fix/hnsw-deleted-topk: live-only search results, batched parallel index build
CI / test (push) Failing after 2s
Details 
Co-Authored-By: Claude Fable 5.1 <[email protected]>
 2026-09-19 13:52:24 -07:00
osobhandClaude Fable 5.1 f507803ec1 feat(agent): build the vector index in parallel by default 
`parallel` joins the agent's default features, so the HNSW bulk build uses the
thread pool: cold index build at 10K records 1152 -> ~380 ms in a same-moment
A/B (the graph is identical either way). Nothing else on the measured paths
changes — ingest, checkpoint, open and steady-state query times are the same
with the feature on or off. Adds rayon to the default dependency set; opt out
with `--no-default-features --features float16,hnsw`.

Harness: `--e2e-only` runs the end-to-end section without the index
benchmarks. Note for anyone comparing numbers: this machine's absolute timings
drifted ~1.5x over a long session, so only same-moment A/B runs are
comparable.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
 2026-09-19 13:52:23 -07:00
osobhandClaude Fable 5.1 8803d0754b ci: lint and test clawhdf5-ann with its parallel feature 
Co-Authored-By: Claude Fable 5.1 <[email protected]>
 2026-09-19 13:44:18 -07:00
osobhandClaude Fable 5.1 42c3872ec9 style(ann): iterate levels directly in the batch entry-point update (clippy) 
Co-Authored-By: Claude Fable 5.1 <[email protected]>
 2026-09-19 13:44:09 -07:00
osobhandClaude Fable 5.1 c19199f3eb perf(ann): batched bulk build, parallel with the parallel feature 
Profiling the build showed 90% of all distance evaluations are in back-link
pruning (40.8M of 44.9M at 10K): every overflow re-runs the diversity
heuristic pairwise over ~max_conn candidates.

The bulk build now inserts in batches: plan every node's neighbours against
the graph as it stood when the batch began (read-only, so plans are
independent), link, then prune each overflowing list once. A node gaining
several back-links in a batch is pruned once rather than once per link, so
this is faster even single-threaded (10K: 1676 -> 1074 ms). With `parallel`,
planning and pruning use rayon (10K: 388 ms; 100K: ~21 s -> 5.9 s on 16
cores). Batches start at one node and are capped at 1/16 of the linked graph
and 512 nodes; a node that raises the top layer gets a batch to itself. The
result is deterministic and identical with or without the feature (one code
path; test compares two builds byte for byte).

Parallelising within a single insert was tried first: 1.45x on 16 cores, tasks
too small. Incremental insert() stays sequential.

Recall on clustered data is unchanged or slightly better; uniform random data
dips slightly (10K, ef=64: 0.474 -> 0.444).

clawhdf5-agent's `parallel` feature now passes through to the index.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
 2026-09-19 13:43:49 -07:00
osobhandClaude Fable 5.1 41db450c92 fix(ann): deletions near the query no longer shrink search results 
search() collected ef candidates, then filtered out soft-deleted nodes, then
took k. When the records nearest a query had been deleted, every candidate was
a tombstone and the search returned fewer than k results — 39 of 40 queries in
the new test, which deletes each query's 40 nearest neighbours.

search_layer takes an optional skip mask: a skipped node is still pushed onto
the candidate queue (a tombstone is a valid waypoint) but never into the
result heap, so the ef result slots hold live nodes only. Build and insert
pass no mask. Recall and speed without deletions are unchanged.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
 2026-09-19 13:37:29 -07:00
393 changed files with 95098 additions and 8776 deletions
Expand all files Collapse all files
.gitea/workflows/ci.yml
+69 -12
View File
@@ -9,32 +9,89 @@ jobs:
runs-on: ubuntu-latest
container: rust:latest
steps:
- uses: actions/checkout@v4
- name: Cache cargo registry/target
uses: actions/cache@v4
with:
path: |
~/.cargo/registry
~/.cargo/git
target
key: ${{ runner.os }}-cargo-${{ hashFiles('**/Cargo.lock') }}
# Plain git rather than actions/checkout: that is a JavaScript action,
# and rust:latest has no `node`, so it failed with exit 127 before any
# code was built — on every push. actions/cache went for the same reason.
- name: Check out
run: |
git init -q .
git remote add origin "${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}.git"
for i in 1 2 3; do git fetch -q --depth 1 origin "${GITHUB_SHA}" && break; sleep 5; done
git checkout -q FETCH_HEAD
- name: Install rustfmt & clippy components
run: rustup component add rustfmt clippy
- name: Install thumbv7em-none-eabihf target
run: rustup target add thumbv7em-none-eabihf
- name: Install wasm32-unknown-unknown target
# ci-test.sh builds the reader and clawhdf5-wasm for the browser.
run: rustup target add wasm32-unknown-unknown
- name: Install Python interop dependencies
# The interop suites used to skip silently when python3/h5py were
# missing, so they never ran in CI. Install them and make a missing
# dependency a failure (CLAWHDF5_REQUIRE_INTEROP below).
run: |
apt-get update
apt-get install -y --no-install-recommends python3 python3-venv
# cmake builds libz-ng-sys for the opt-in `fast-deflate` (zlib-ng)
# steps in ci-test.sh; rust:latest does not ship it. The default
# build (pure-Rust zlib-rs) does not need it.
# hdf5-tools: h5ls/h5stat/h5dump/h5diff, which the h5rs
# (clawhdf5-tools) interop tests compare against.
apt-get install -y --no-install-recommends python3 python3-venv cmake hdf5-tools
python3 -m venv /opt/interop
/opt/interop/bin/pip install --no-cache-dir h5py numpy netCDF4 xarray
# maturin + pytest: ci-test.sh builds the Python package
# (crates/clawhdf5-py) and runs its tests against h5py.
/opt/interop/bin/pip install --no-cache-dir h5py numpy netCDF4 xarray hdf5plugin maturin pytest
echo "/opt/interop/bin" >> "$GITHUB_PATH"
- name: Show interop library versions
run: python3 -c "import h5py, netCDF4; print('h5py', h5py.__version__, 'HDF5', h5py.version.hdf5_version, 'netCDF4', netCDF4.__version__)"
# h5dump's version too: the h5rs dump test requires its exact output
# (checked against Debian's 1.14.5 in rust:latest and 1.14.6).
run: |
/opt/interop/bin/python -c "import h5py, netCDF4, hdf5plugin; print('h5py', h5py.__version__, 'HDF5', h5py.version.hdf5_version, 'netCDF4', netCDF4.__version__, 'hdf5plugin', hdf5plugin.version)"
h5dump --version
- name: Run CI script
env:
# Name the interpreter outright rather than relying on $GITHUB_PATH
# reaching the test processes: if `python3` resolved to the system
# one instead of the venv, every interop suite would skip.
# CLAWHDF5_REQUIRE_INTEROP turns that skip into a failure, so the
# two together mean the suites either run or the build goes red.
CLAWHDF5_PYTHON: /opt/interop/bin/python
CLAWHDF5_REQUIRE_INTEROP: "1"
run: bash scripts/ci-test.sh
test-arm64:
# The aarch64 kernels in clawhdf5-accel — NEON `dot_i8`, including the
# SDOT path, and the f32 NEON kernels — are cfg'd out on x86, so the job
# above never compiles, lints or tests them.
#
# `linux_arm64` is served by two runners that execute differently:
# vision-01 runs steps on the host (Rust already installed) and vision-02
# runs them in docker.gitea.com/runner-images. So the steps work in both:
# no `container:`, no JavaScript actions (they are fetched from GitHub,
# which not every runner reliably reaches), and an explicit `+stable`
# toolchain rather than whatever a host happens to default to.
runs-on: linux_arm64
env:
CARGO_NET_RETRY: "10"
CARGO_TERM_COLOR: always
steps:
- name: Check out
run: |
git init -q .
git remote add origin "${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}.git"
for i in 1 2 3; do git fetch -q --depth 1 origin "${GITHUB_SHA}" && break; sleep 5; done
git checkout -q FETCH_HEAD
- name: Rust stable
run: |
export PATH="$HOME/.cargo/bin:$PATH"
command -v rustup >/dev/null || curl -sSf --retry 5 https://sh.rustup.rs | sh -s -- -y --profile minimal --default-toolchain none
rustup toolchain install stable --profile minimal --component clippy
echo "$HOME/.cargo/bin" >> "$GITHUB_PATH"
- name: Confirm aarch64
run: |
test "$(uname -m)" = aarch64
if grep -q asimddp /proc/cpuinfo; then echo "dot-product extension present: SDOT kernel runs"; else echo "no dot-product extension: plain NEON kernel runs"; fi
- name: Clippy (aarch64 kernels)
run: cargo +stable clippy -p clawhdf5-accel --all-targets -- -D warnings
- name: Test
run: cargo +stable test -p clawhdf5-accel -p clawhdf5-ann -p clawhdf5-format
.gitea/workflows/conformance.yml
+56
View File
@@ -0,0 +1,56 @@
name: Conformance
# Nightly: read every file of the pinned public HDF5 corpora with clawhdf5 and
# with h5py/libhdf5 and compare (conformance/run.sh; CONFORMANCE.md explains
# the method). Fails on any panic, hang, crash or out-of-memory in clawhdf5,
# and when the ok count drops below conformance/baseline.json or a file the
# baseline lists as ok stops being ok. The report is printed into the job log;
# nothing is uploaded (artifact actions are JavaScript, which rust:latest
# cannot run — see CLAUDE.md).
on:
schedule:
- cron: "17 3 * * *"
workflow_dispatch:
jobs:
conformance:
runs-on: ubuntu-latest
container: rust:latest
timeout-minutes: 60
env:
CARGO_NET_RETRY: "10"
steps:
# Plain git, not actions/checkout (a JavaScript action; see ci.yml).
- name: Check out
run: |
git init -q .
git remote add origin "${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}.git"
for i in 1 2 3; do git fetch -q --depth 1 origin "${GITHUB_SHA}" && break; sleep 5; done
git checkout -q FETCH_HEAD
- name: Install h5py, h5dump and the probe's codec libraries
# hdf5-tools: h5dump for the CVE-corpus comparison. libaec-dev and
# pkg-config: the probe builds clawhdf5-format with `szip` (the core
# crates' default build needs neither).
run: |
apt-get update
apt-get install -y --no-install-recommends python3 python3-venv hdf5-tools libaec-dev pkg-config
python3 -m venv /opt/conformance
/opt/conformance/bin/pip install --no-cache-dir -r conformance/requirements.txt
/opt/conformance/bin/python -c "import h5py, hdf5plugin; print('h5py', h5py.__version__, 'HDF5', h5py.version.hdf5_version, 'hdf5plugin', hdf5plugin.version)"
h5dump --version
- name: Probe unit tests
run: cargo test --release --manifest-path conformance/probe/Cargo.toml
env:
CARGO_TARGET_DIR: conformance/.cache/target
- name: Sweep
# The corpora come from GitHub (pinned commits, conformance/corpus.txt),
# so this job needs a runner that reaches github.com.
env:
CLAWHDF5_PYTHON: /opt/conformance/bin/python
run: bash conformance/run.sh
- name: Report
if: always()
run: |
if [ -f CONFORMANCE.md ]; then cat CONFORMANCE.md; else echo "no report was generated"; fi
if [ -f conformance/.cache/results/summary.md ]; then
echo; echo "---- per-file detail (conformance/.cache/results/summary.md) ----"
cat conformance/.cache/results/summary.md
fi
.gitignore
+3
View File
@@ -4,3 +4,6 @@ benchmarks/longmemeval/*.json
# Local model weights (MiniLM etc.) — large, not committed
weights/
.venv
__pycache__/
.pytest_cache/
BENCHMARKS.md
+1396 -140
View File
File diff suppressed because it is too large Load Diff
CHANGELOG.md
+2292
View File
File diff suppressed because it is too large Load Diff
CLAUDE.md
+151 -12
View File
@@ -1,43 +1,66 @@
# clawhdf5
## Purpose
Pure-Rust HDF5 format implementation with HNSW vector search, WAL-backed persistence, agent memory storage, and GPU-accelerated I/O. Used by ZeroClaw as its persistent memory and knowledge graph backend.
Pure-Rust HDF5 format implementation with HNSW vector search, WAL-backed persistence, agent memory storage, and GPU-accelerated vector search. A standalone library. Its one verified consumer is ClawBrainHub (`.brain` files); no agent framework integrates it (OpenClaw and ZeroClaw claims were withdrawn on 2026-09-25 — neither was ever true).
## Architecture
Cargo workspace with 16 crates under `crates/` (plus `libaec-sys`, an internal FFI bindings crate for the optional `szip` feature):
Cargo workspace with 19 crates under `crates/` (plus `libaec-sys`, an internal FFI bindings crate for the optional `szip` feature):
| Crate | Role |
|-------|------|
| `clawhdf5-format` | HDF5 binary spec parser (superblock, B-tree, heap) — also holds shared type definitions and physical constants |
| `clawhdf5-io` | Read/write implementation |
| `clawhdf5-filters` | Compression filters (gzip, LZ4, Zstd, Blosc) |
| `clawhdf5-filters` | Deflate backends (zlib-rs, zlib-ng, Apple Compression); the HDF5 filter pipeline, the filter registry (`clawhdf5_format::filter_registry`) and the other codecs (LZ4, Zstd, SZIP, N-Bit, scale-offset, pcodec, and the pure-Rust plugin filters LZF, bitshuffle, bzip2, Blosc 1, and Blosc2 and ZFP read-only) live in `clawhdf5-format`. |
| `clawhdf5-derive` | Proc-macro derive for HDF5-serializable structs |
| `clawhdf5` | Main facade crate |
| `clawhdf5-netcdf4` | NetCDF-4 compatibility layer |
| `clawhdf5-ann` | HNSW approximate nearest-neighbor vector index |
| `clawhdf5-agent` | Agent memory, session history, knowledge graph storage |
| `clawhdf5-gpu` | GPU-accelerated I/O via wgpu (hand-written WGSL compute shaders) |
| `clawhdf5-gpu` | GPU vector distance computation via wgpu (hand-written WGSL compute shaders) — not dataset I/O |
| `clawhdf5-accel` | CPU SIMD acceleration path |
| `clawhdf5-migrate` | Schema migration engine |
| `clawhdf5-migrate` | SQLite → HDF5 agent-memory migration |
| `clawhdf5-android` | Android JNI bindings |
| `clawhdf5-cli` | Command-line interface |
| `clawhdf5-cli` | Command-line interface (agent memory) |
| `clawhdf5-tools` | `h5rs`: pure-Rust HDF5 tools — `ls`, `dump` (DDL / hdf5-json), `stat`, `diff`, `check` (structural + checksum validator) |
| `clawhdf5-napi` | Node.js native addon bindings |
| `clawhdf5-py` | PyO3 Python bindings |
| `clawhdf5-wasm` | WebAssembly (wasm-bindgen) reader for the browser; demo in `examples/wasm-viewer/` |
| `clawhdf5-remote` | Remote files: `open_url` over HTTP(S) range requests and object stores (`object_store`: S3, GCS, Azure) through a mandatory block cache (`BlockCache`) |
| `clawhdf5-bench` | Benchmark suite |
## Key Features
- Zero-dependency HDF5 read/write (no libhdf5 C library required)
- Zero-C-dependency HDF5 read/write: no libhdf5, and deflate defaults to
pure-Rust zlib-rs (`fast-deflate` opts into zlib-ng, which needs cmake).
`ci-test.sh` fails if a C-building crate enters the core crates' default
tree. flate2 must keep `runtime_detection` with zlib-rs — without it zlib-rs
loses SIMD and inflates 3.5x slower. MSRV is 1.92 (`rust-version`, checked
in CI).
- HNSW vector index for semantic similarity search over agent memories — the
`clawhdf5-agent` `hnsw` feature is **on by default**, so `hybrid_search` uses
the approximate `clawhdf5-ann` index for the vector stage (the index mirrors
the cache and self-heals on drift). Build the agent with
`--no-default-features --features float16` to force the exact linear cosine scan.
The agent's `parallel` feature (also default) builds the index on a thread
pool; the graph is identical with or without it.
The index uses the HNSW paper's diversity heuristic for neighbour selection
(plain closest-M capped recall on clustered data: 0.31 recall@10 at 100K). Its
graph is saved to `<store>.h5.ann` at each checkpoint and reloaded by `open()`
(tied to the checkpoint by a generation id; stale/damaged sidecars are
ignored and the index rebuilt). `hybrid_search` keeps one incremental BM25
ignored and the index rebuilt). `MemoryConfig::quantized_index` (**on by
default** for new stores, persisted; stores predating the setting load as
`false` and keep their f32 index — guarded by
`tests/fixtures/store_v2_5_0.h5`; CLI opt-out is `create --f32-index`)
stores the index's own copy of the embeddings as `i8`,
which roughly halves a loaded store's memory (2.72x -> 1.74x the raw vectors
at 100K); because quantised distances are approximate and `ef` cannot
compensate, the query path then re-scores the candidate pool against the
exact embeddings, which holds recall at the f32 index's level. It is also
faster at equal recall: 1.63x the QPS on x86-64 (AVX2) and 1.18x on a
Raspberry Pi 5 (`clawhdf5_accel::dot_i8`, NEON `SDOT` via inline asm since
the intrinsic is unstable; plain NEON on pre-dotprod cores). The aarch64
code is `cfg`'d out on x86, so x86 CI never compiles or lints it — test it
on real ARM (`rpivision02`, 10.0.2.3, is a Pi 5). `hybrid_search` keeps one incremental BM25
index for the life of the store and never writes the store: Hebbian
activation boosts are persisted by the next checkpoint (or on drop), not per
query. Measure any search-path change with
@@ -67,8 +90,51 @@ Cargo workspace with 16 crates under `crates/` (plus `libaec-sys`, an internal F
`export` do). An unreadable WAL (torn header, bad magic) is quarantined to
`<store>.h5.wal.corrupt-<ts>` rather than blocking `open()`; a WAL with an
unknown *newer* version still fails and is left untouched.
- `MemoryConfig::compression` uses deflate by default; enable the agent's
`zstd` feature to compress embeddings with Zstd instead (links libzstd).
- `MemoryConfig::float16` (**on by default** for new stores, persisted;
existing stores keep their recorded `false` — guarded by the v2.5.0
fixture in `tests/float16_store.rs`; CLI opt-out is `create --f32`) writes
`/memory/embeddings` as IEEE half precision (48% smaller file at 100K;
LongMemEval with real MiniLM embeddings identical to f32).
`MemoryCache::half_precision` rounds each embedding as it enters the cache (push, update, WAL replay, and on load of a store still
`f32` on disk), so memory and file agree bit for bit; the conversions live
in `clawhdf5_format::float16` and must stay the single implementation.
Values beyond ±65504 are `MemoryError::InvalidEntry`. Interop: every file
must open in h5py — `f32` datasets and empty datasets did not until
2026-09-23 (see `docs/known-issues.md`); the agent's `h5py_interop` test
guards a whole store.
- `HDF5Memory::search(query_emb, text, &SearchOptions)` is the full search
path: optional source-channel filter (applied before ranking; exact scan of
the allowed records whenever cheaper than `pool × M` index distance
evaluations, and as the fallback when the pool comes back short), fusion,
activation scaling, optional re-ranking and confidence rejection.
`hybrid_search`/`hybrid_search_with` are thin wrappers; `ClawhdfBackend`
(the `openclaw` module) is `search` with re-rank + confidence on.
- **OpenClaw is not supported** (decided 2026-09-25): clawhdf5 is not an
OpenClaw memory plugin and never was — the old `memory.backend = "clawhdf5"`
config was never valid. Don't reintroduce OpenClaw claims; `docs/openclaw.md`
records what a real plugin would need.
- **ZeroClaw does not use clawhdf5** (checked 2026-09-25 against upstream
v0.8.5 and the `osobh/zeroclaw` fork, and their full history): no
`clawhdf5` feature or backend exists; ZeroClaw's memory backends are
sqlite/lucid/postgres/qdrant/markdown/none behind its own `Memory` trait.
`clawhdf5-migrate`'s default SQLite layout (`memory_chunks`, `sessions`,
`entities`, `relations`) is not ZeroClaw's schema either (ZeroClaw's is a
`memories` table). Don't reintroduce integration claims without an
integration and a test against the real consumer. Measure changes with
`search_harness --options-study`.
- `MemoryConfig::compression` is off by default; when on, embeddings are
deflate-compressed, or Zstd with the agent's `zstd` feature (links libzstd).
- Signed checkpoints (`clawhdf5-agent` `signing` module): with
`HDF5Memory::set_signing_key` every checkpoint stores an Ed25519-signed
manifest (SHA-256 per record in a Merkle tree + settings/sessions/graph
hashes; per-record hashes in `/integrity/record_hashes`);
`HDF5Memory::verify(path, &pk)` locates edits. The hashes must cover exactly
what the file persists in the form the loader returns it (strings lose
trailing NULs; an empty WAL mark is not written) or untouched stores stop
verifying — `tests/signed_store.rs` round-trips awkward strings. The key is
never persisted; a signed store refuses to checkpoint without it
(`MemoryError::SigningKeyRequired`, and `MemoryError` is `#[non_exhaustive]`).
WAL entries after the checkpoint are not covered.
- `Dataset::verify_provenance()` (clawhdf5 facade, `provenance` feature, on by
default) recomputes a dataset's SHA-256 and compares it against the
`_provenance_sha256` attribute written automatically on save when
@@ -85,7 +151,44 @@ Cargo workspace with 16 crates under `crates/` (plus `libaec-sys`, an internal F
Alerts never block a save — drain them with `HDF5Memory::take_anomaly_alerts`.
`MemorySource` for this bookkeeping is inferred from the caller-supplied
`source_channel` string (a heuristic, not an authenticated trust boundary).
- GPU-accelerated batch I/O for large dataset processing
- In-place modification: `clawhdf5::FileEditor` (`crates/clawhdf5/src/edit/`)
overwrites values, grows and shrinks chunked datasets (every chunk index,
version-2 B-trees included) and sets attributes (compact and dense
storage) in existing files (h5py- or clawhdf5-written) without rewriting
them, changing indexes and heaps as libhdf5 does (index shapes and heap
bookkeeping are compared with libhdf5's in the tests); space an edit
frees is reused by later edits of the same editor. Anything it cannot do
safely is `Error::Unsupported` before any write (limits in
`docs/known-issues.md`). Test changes with
`cargo test -p clawhdf5-tools --test edit_interop --test
edit_coverage_interop` (h5py, h5dump, `h5rs check`, structure comparisons
with libhdf5; libhdf5 sources for the algorithms are at
github.com/HDFGroup/hdf5, tag `hdf5_1_14_6`).
- Remote files (`clawhdf5-remote`, range-read milestone M3 of
`docs/design/range-reads.md`): `open_url("http://…")` gives a
`clawhdf5::File` over `File::open_storage`, read through `BlockCache`
(1 MiB blocks, LRU byte budget, per-block in-flight dedup across threads,
runs coalesced into parallel requests). `HttpStorage` pins the file by
ETag/Last-Modified and length (a change is `RemoteError::FileChanged`),
refuses servers that ignore `Range` unless a full download is allowed,
and retries transient failures. `ObjectStoreStorage` (feature
`object-store`, pure Rust) runs each read on a small owned tokio
runtime and waits on a channel, so it works from any thread, including
inside `spawn_blocking` or another runtime. Default build is plain HTTP with
no C; `https` (rustls + ring) and `s3`/`gcs`/`azure` (aws-lc-rs) are
opt-in. Tests run a std-only HTTP server
(`tests/common/server.rs`, also the `range_server` example);
`CLAWHDF5_REMOTE_CORPUS=conformance/.cache/corpus` compares every corpus
file over HTTP with `File::open`.
- GPU-accelerated vector distance computation (`clawhdf5-gpu`, wgpu); HDF5 I/O itself is CPU-only
- Browser: `clawhdf5-wasm` (wasm-bindgen, read-only, file held in memory;
no Zstd/SZIP since they link C) and the `examples/wasm-viewer/` page.
`examples/wasm-viewer/test/run.sh` builds the package (needs the
`wasm-bindgen` CLI at the crate's exact version) and tests it under Node
and headless Chromium (a Playwright download in `~/.cache/ms-playwright`
on tank); the CI container has neither, so CI runs the native
`clawhdf5-wasm` `h5py_interop` test on the same fixture. Size numbers are
in the example's README.
- Python and Node.js bindings for cross-language use
- NetCDF-4 compatibility for scientific data interop
@@ -101,12 +204,40 @@ cargo build --release
cargo test --workspace
```
### CI
`.gitea/workflows/ci.yml` has two jobs, both green as of 2026-09-22:
- **`test`** (`ubuntu-latest`, in `rust:latest`) runs `scripts/ci-test.sh` with
the h5py/netCDF4 interop suites required (`CLAWHDF5_REQUIRE_INTEROP=1`).
Served by the `tank` and `architect` runners.
- **`test-arm64`** (`linux_arm64`) lints and tests the aarch64 code — the NEON
kernels are `cfg`'d out on x86, so this is the only place they are built.
Served by `vision-01` (host mode) and `vision-02` (Docker), so steps must
work in both.
Keep workflows free of JavaScript actions (`actions/checkout`, `actions/cache`,
…): `rust:latest` has no `node`, and not every runner reaches GitHub, where
they are fetched from. Check out with plain `git` instead. The `test` job
installs `cmake` for the opt-in `fast-deflate` (zlib-ng) steps; the default
build needs no C toolchain, so `test-arm64` does not.
All runners are on `gitea-runner` 3.5.0, from `docker.gitea.com/act_runner`
— `gitea/act_runner:latest` on Docker Hub is frozen at 0.6.1.
### CLI
```bash
cargo run -p clawhdf5-cli -- --help
# create, save, search, recall, stats, flush-wal, agents-md, export, snapshot subcommands
```
### HDF5 tools (`h5rs`, crate `clawhdf5-tools`)
```bash
cargo run -p clawhdf5-tools -- ls -r file.h5 # also dump [--json], stat, diff, check
bash scripts/h5rs-fuzz.sh # every subcommand over the CVE corpus: no panic/crash/hang
bash scripts/h5rs-check-ok-files.sh --data # check passes every fully-read conformance file
```
Its interop tests compare against h5ls/h5stat/h5dump/h5diff (Debian
`hdf5-tools`, installed in CI); `dump` must stay byte-identical to h5dump on
the test files.
### Python bindings
```bash
cd crates/clawhdf5-py
@@ -115,4 +246,12 @@ python -c "import clawhdf5; print(clawhdf5.__version__)"
```
## Integration
ZeroClaw imports this as a Cargo feature (`clawhdf5` feature flag) to persist agent memory with HNSW vector search for context retrieval.
- **ClawBrainHub** (`clawverse/clawbrainhub` on git.redclaw.dev) is the one
verified consumer: `cbh-core` reads and writes `.brain` files through the
facade (`File`, `FileBuilder`, `AttrValue`, `Selection`), `cbh-scanner`
uses the facade, and `cbh-cli` uses `clawhdf5_agent::bm25::BM25Index`. It
depends on this repo by path (`../clawhdf5`), so it builds against whatever
is checked out — changes to those APIs reach it directly. Verified
2026-09-25 against main: builds, and its 204 tests pass.
- OpenClaw and ZeroClaw were both described as consumers; neither integrates
clawhdf5 (see Key Features and `docs/openclaw.md`).
CONFORMANCE.md
+291
View File
@@ -0,0 +1,291 @@
# clawhdf5 conformance report
Every HDF5 file of eight public corpora (pinned by commit) is read twice — by
clawhdf5 (`conformance/probe`, the same `clawhdf5-format` calls the facade
makes) and by h5py/libhdf5 (`conformance/ref.py`) — and the two readings are
compared object by object: the set of hard-linked objects, each dataset's and
attribute's shape, and a SHA-256 of its values in a canonical encoding. The
CVE corpus is also run through `h5dump`. Each side runs under a timeout and an
address-space limit, so a hang, crash or runaway allocation is recorded, not
fatal. This file is generated by `conformance/run.sh`; do not edit it by hand.
## Run
| | |
|---|---|
| date | 2026-09-27 00:34 UTC |
| clawhdf5 commit | `f37e7ae3263277319dba4bc39be5397194eb00c3` |
| machine | `tank`: AMD Ryzen 7 7800X3D 8-Core Processor, 16 CPUs, 61 GiB, Linux 7.0.0-34-generic x86_64 |
| command | `conformance/run.sh --no-fetch --update-baseline` |
| rustc | rustc 1.98.1 (48a229cea 2026-09-01) |
| reference | h5py 3.16.0, HDF5 2.0.0, numpy 2.5.3, hdf5plugin 7.1.0, Python 3.14.4 |
| h5dump | Version 1.14.6 (CVE corpus only) |
| limits | 20 s timeout (SIGKILL), 4096 MiB address space, per process; 16 files in parallel |
| runtime | 21 s probing + comparing (0 s fetch/build before it) |
## Results
A file's class is the first that applies:
- **panic / hang / crash / oom** — clawhdf5 panicked (caught per object or not), hit the timeout, died on a signal, or failed an allocation. The CI gate fails on any of these.
- **h5py-cannot-read** — libhdf5 could not open the file (or itself crashed or hung). Nothing to compare against; most are the deliberately malformed CVE reproducers.
- **our-error** — clawhdf5 returned an error for something h5py reads.
- **mismatch** — both read it, but the shapes, values, object set or attribute set differ.
- **ok** — every object h5py reads, clawhdf5 reads identically.
| corpus | files | ok | our-error | mismatch | h5py-cannot-read | panic | hang | crash | oom |
|---|---|---|---|---|---|---|---|---|---|
| NCAS-CMS_pyfive | 33 | 32 | 0 | 1 | 0 | 0 | 0 | 0 | 0 |
| cve_hdf5 | 147 | 113 | 2 | 0 | 32 | 0 | 0 | 0 | 0 |
| h5py_data | 4 | 4 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| hdf5 | 466 | 404 | 1 | 1 | 60 | 0 | 0 | 0 | 0 |
| netcdf-c | 20 | 20 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| netcdf4-python | 18 | 18 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| usnistgov_h5wasm | 5 | 5 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| xarray-data | 4 | 4 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| **all** | **697** | **600** | **3** | **2** | **92** | **0** | **0** | **0** | **0** |
2 of the 2 mismatches are a known h5py bug, not ours (see *Known not-our-bug*).
3 of the 3 our-errors are corrupt data that HDF5 2.0 reads only through a bug and clawhdf5 refuses (see *Known not-our-bug*).
Corpora (fetched by `conformance/fetch-corpus.sh` into the gitignored `conformance/.cache/`):
| corpus | source | commit |
|---|---|---|
| hdf5 | https://github.com/HDFGroup/hdf5 | `a3cf1ea82cc7` |
| cve_hdf5 | https://github.com/HDFGroup/cve_hdf5 | `3fd1f5ae3869` |
| netcdf-c | https://github.com/Unidata/netcdf-c | `beb7b9585273` |
| NCAS-CMS_pyfive | https://github.com/NCAS-CMS/pyfive | `8cf07b874913` |
| usnistgov_h5wasm | https://github.com/usnistgov/h5wasm | `02f6336527d2` |
| netcdf4-python | https://github.com/Unidata/netcdf4-python | `6e67576d39ae` |
| xarray-data | https://github.com/pydata/xarray-data | `a35297e9da2c` |
| h5py_data | https://github.com/h5py/h5py (`h5py/tests/data_files`) | `b2f0347c4200` |
## Panics, hangs, crashes, out-of-memory
None.
## Our-error root causes
Grouped by normalised error message. *files* counts files whose class this cause affects.
| files | objects | error | examples |
|---:|---:|---|---|
| 3 | 3 | `ChunkedReadError("…")` | `cve_hdf5/cvefiles/cve-2025-2308.h5`, `cve_hdf5/cvefiles/cve-2025-44904.h5`, `hdf5/test/testfiles/bad_nbit_parms_walk.h5` |
## Mismatch root causes
| files | objects | cause | examples |
|---:|---:|---|---|
| 1 | 1 | `attr-values: ours=vlen(>u8) h5py=object layout=- filters=-` | `NCAS-CMS_pyfive/tests/data/attr_datatypes.hdf5` |
| 1 | 1 | `values: ours=vlen({r:>f4,i:>f4}8) h5py=object layout=contiguous filters=-` | `hdf5/tools/test/testfiles/tcomplex_be.h5` |
## CVE corpus: clawhdf5 vs h5dump vs h5py
The 147 files of [HDFGroup/cve_hdf5](https://github.com/HDFGroup/cve_hdf5) — reproducers for
published libhdf5 CVEs and fuzzer finds. *read* = produced output (possibly with per-object
errors), *error* = refused cleanly. h5dump exits non-zero on any error anywhere in a file, so
its read/error split is not comparable with the other two rows; the panic, crash, hang and oom
columns are.
| tool | read | error | panic | crash | hang | oom |
|---|---:|---:|---:|---:|---:|---:|
| clawhdf5 | 121 | 26 | 0 | 0 | 0 | 0 |
| h5dump 1.14.6 | 16 | 129 | 0 | 2 | 0 | 0 |
| h5py 3.16.0 / HDF5 2.0.0 | 115 | 31 | 0 | 1 | 0 | 0 |
<details><summary>Per-file outcomes</summary>
| file | h5dump | h5py | clawhdf5 | class |
|---|---|---|---|---|
| cvefiles/cve-2016-4330.h5 | error exit | read 3 obj, 1 errors | read 3 obj, 1 errors | ok |
| cvefiles/cve-2016-4331.h5 | error exit | read 25 obj, 1 errors | read 25 obj, 1 errors | ok |
| cvefiles/cve-2016-4332-mtime-new.h5 | error exit | read 25 obj, 1 errors | read 25 obj, 1 errors | ok |
| cvefiles/cve-2016-4332-mtime.h5 | error exit | read 4 obj, 3 errors | read 4 obj, 3 errors | ok |
| cvefiles/cve-2016-4332-stab.h5 | error exit | open error | read 1 obj, 1 errors | h5py-cannot-read |
| cvefiles/cve-2016-4333.h5 | error exit | read 3 obj, 1 errors | read 3 obj, 1 errors | ok |
| cvefiles/cve-2017-17505.h5 | error exit | read 2 obj, 1 errors | read 2 obj, 1 errors | ok |
| cvefiles/cve-2017-17506.h5 | error exit | read 2 obj, 1 errors | read 2 obj, 1 errors | ok |
| cvefiles/cve-2017-17507.h5 | error exit | read 2 obj, 1 errors | read 2 obj, 1 errors | ok |
| cvefiles/cve-2017-17508.h5 | error exit | read 2 obj, 1 errors | read 2 obj, 1 errors | ok |
| cvefiles/cve-2017-17509.h5 | error exit | read 2 obj, 1 errors | read 2 obj, 1 errors | ok |
| cvefiles/cve-2018-11202.h5 | error exit | read 2 obj, 1 errors | read 2 obj, 1 errors | ok |
| cvefiles/cve-2018-11203.h5 | error exit | read 2 obj, 1 errors | read 2 obj, 1 errors | ok |
| cvefiles/cve-2018-11204.h5 | error exit | read 2 obj, 1 errors | read 2 obj, 1 errors | ok |
| cvefiles/cve-2018-11205.h5 | error exit | read 2 obj, 1 errors | read 2 obj, 1 errors | ok |
| cvefiles/cve-2018-11206-new.h5 | error exit | read 3 obj, 1 errors | read 3 obj, 1 errors | ok |
| cvefiles/cve-2018-11206-old.h5 | error exit | read 3 obj, 1 errors | read 3 obj, 1 errors | ok |
| cvefiles/cve-2018-11207.h5 | error exit | read 2 obj, 1 errors | read 2 obj, 1 errors | ok |
| cvefiles/cve-2018-13866.h5 | error exit | open error | open error | h5py-cannot-read |
| cvefiles/cve-2018-13867.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
| cvefiles/cve-2018-13868.h5 | error exit | read 3 obj, 1 errors | read 3 obj, 1 errors | ok |
| cvefiles/cve-2018-13869.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
| cvefiles/cve-2018-13870.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
| cvefiles/cve-2018-13871.h5 | error exit | read 2 obj | read 2 obj | ok |
| cvefiles/cve-2018-13872.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
| cvefiles/cve-2018-13873.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
| cvefiles/cve-2018-13874.h5 | error exit | open error | open error | h5py-cannot-read |
| cvefiles/cve-2018-13875.h5 | error exit | read 3 obj, 1 errors | read 3 obj, 1 errors | ok |
| cvefiles/cve-2018-13876.h5 | error exit | open error | open error | h5py-cannot-read |
| cvefiles/cve-2018-14031.h5 | error exit | read 3 obj, 1 errors | read 3 obj, 1 errors | ok |
| cvefiles/cve-2018-14033.h5 | error exit | read 3 obj, 1 errors | read 3 obj, 1 errors | ok |
| cvefiles/cve-2018-14034.h5 | error exit | read 1 obj, 2 errors | read 1 obj | ok |
| cvefiles/cve-2018-14035.h5 | error exit | read 3 obj, 1 errors | read 3 obj, 1 errors | ok |
| cvefiles/cve-2018-14460.h5 | error exit | read 3 obj, 2 errors | read 3 obj, 2 errors | ok |
| cvefiles/cve-2018-15671.h5 | ok | read 1 obj | read 1 obj | ok |
| cvefiles/cve-2018-15672.h5 | error exit | read 2 obj, 1 errors | read 2 obj, 1 errors | ok |
| cvefiles/cve-2018-16438.h5 | error exit | read 1 obj, 1 errors | read 1 obj | ok |
| cvefiles/cve-2018-17233.h5 | error exit | read 6 obj, 1 errors | read 6 obj, 1 errors | ok |
| cvefiles/cve-2018-17234.h5 | error exit | read 6 obj, 1 errors | read 6 obj, 1 errors | ok |
| cvefiles/cve-2018-17237.h5 | error exit | read 2 obj, 1 errors | read 2 obj, 1 errors | ok |
| cvefiles/cve-2018-17432.h5 | error exit | read 3 obj, 1 errors | read 3 obj, 1 errors | ok |
| cvefiles/cve-2018-17433 | error exit | open error | open error | h5py-cannot-read |
| cvefiles/cve-2018-17434.h5 | error exit | read 3 obj, 1 errors | read 3 obj, 1 errors | ok |
| cvefiles/cve-2018-17435.h5 | error exit | read 3 obj, 1 errors | read 3 obj, 1 errors | ok |
| cvefiles/cve-2018-17436 | error exit | open error | open error | h5py-cannot-read |
| cvefiles/cve-2018-17437.h5 | error exit | read 3 obj, 1 errors | read 3 obj, 1 errors | ok |
| cvefiles/cve-2018-17438 | error exit | read 3 obj, 1 errors | read 3 obj, 1 errors | ok |
| cvefiles/cve-2018-17439 | error exit | read 3 obj, 1 errors | read 3 obj, 1 errors | ok |
| cvefiles/cve-2019-8396.h5 | error exit | read 3 obj, 2 errors | read 3 obj, 2 errors | ok |
| cvefiles/cve-2019-8397.h5 | error exit | read 2 obj, 1 errors | read 2 obj, 1 errors | ok |
| cvefiles/cve-2019-8398.h5 | error exit | read 2 obj, 1 errors | read 2 obj, 1 errors | ok |
| cvefiles/cve-2019-9151.h5 | error exit | read 3 obj, 1 errors | read 3 obj, 2 errors | ok |
| cvefiles/cve-2019-9152.h5 | error exit | read 3 obj, 1 errors | read 3 obj, 1 errors | ok |
| cvefiles/cve-2020-10809 | error exit | open error | open error | h5py-cannot-read |
| cvefiles/cve-2020-10810.h5 | error exit | open error | open error | h5py-cannot-read |
| cvefiles/cve-2020-10811.h5 | error exit | read 25 obj, 1 errors | read 25 obj, 1 errors | ok |
| cvefiles/cve-2020-10812.h5 | error exit | open error | open error | h5py-cannot-read |
| cvefiles/cve-2020-18232.h5 | error exit | read 3 obj, 2 errors | read 3 obj, 2 errors | ok |
| cvefiles/cve-2020-18494.h5 | ok | read 2 obj | read 2 obj | ok |
| cvefiles/cve-2021-36977.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
| cvefiles/cve-2021-37501.h5 | error exit | read 18 obj, 1 errors | read 18 obj, 1 errors | ok |
| cvefiles/cve-2021-45829.h5 | error exit | read 1 obj, 2 errors | read 1 obj, 1 errors | ok |
| cvefiles/cve-2021-45830.h5 | error exit | open error | open error | h5py-cannot-read |
| cvefiles/cve-2021-45833.h5 | error exit | read 2 obj, 1 errors | read 2 obj, 1 errors | ok |
| cvefiles/cve-2021-46242.h5 | error exit | open error | read 1 obj, 1 errors | h5py-cannot-read |
| cvefiles/cve-2021-46243.h5 | error exit | read 2 obj, 1 errors | read 2 obj, 1 errors | ok |
| cvefiles/cve-2021-46244.h5 | error exit | read 2 obj, 1 errors | read 2 obj, 1 errors | ok |
| cvefiles/cve-2024-29157.h5 | error exit | read 4 obj, 7 errors | read 4 obj, 7 errors | ok |
| cvefiles/cve-2024-29158.h5 | ok | read 3 obj, 1 errors | read 3 obj, 1 errors | ok |
| cvefiles/cve-2024-29159.h5 | error exit | read 2 obj, 1 errors | read 2 obj, 1 errors | ok |
| cvefiles/cve-2024-29160.h5 | error exit | read 4 obj, 1 errors | read 4 obj, 1 errors | ok |
| cvefiles/cve-2024-29161.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
| cvefiles/cve-2024-29162.h5 | error exit | read 17 obj, 4 errors | read 17 obj, 4 errors | ok |
| cvefiles/cve-2024-29163.h5 | error exit | read 7 obj, 1 errors | read 7 obj, 1 errors | ok |
| cvefiles/cve-2024-29164.h5 | ok | read 3 obj | read 3 obj | ok |
| cvefiles/cve-2024-29165.h5 | error exit | read 2 obj, 1 errors | read 2 obj, 1 errors | ok |
| cvefiles/cve-2024-29166.h5 | error exit | read 17 obj, 2 errors | read 17 obj | ok |
| cvefiles/cve-2024-32605.h5 | ok | read 6 obj, 1 errors | read 6 obj, 1 errors | ok |
| cvefiles/cve-2024-32606.h5 | error exit | read 2 obj, 1 errors | read 2 obj, 1 errors | ok |
| cvefiles/cve-2024-32607-1.h5 | ok | read 10 obj | read 10 obj | ok |
| cvefiles/cve-2024-32607-2.h5 | error exit | read 9 obj, 1 errors | read 9 obj, 1 errors | ok |
| cvefiles/cve-2024-32608.h5 | error exit | read 6 obj, 1 errors | read 6 obj, 1 errors | ok |
| cvefiles/cve-2024-32609.h5 | error exit | SIGSEGV | read 3 obj, 1 errors | h5py-cannot-read |
| cvefiles/cve-2024-32610.h5 | error exit | read 2 obj, 1 errors | read 2 obj, 1 errors | ok |
| cvefiles/cve-2024-32611.h5 | ok | read 6 obj | read 6 obj | ok |
| cvefiles/cve-2024-32612.h5 | ok | read 3 obj | read 3 obj | ok |
| cvefiles/cve-2024-32613.h5 | error exit | read 7 obj, 1 errors | read 7 obj, 1 errors | ok |
| cvefiles/cve-2024-32614.h5 | error exit | read 25 obj, 2 errors | read 25 obj, 2 errors | ok |
| cvefiles/cve-2024-32615.h5 | error exit | read 4 obj, 1 errors | read 4 obj, 1 errors | ok |
| cvefiles/cve-2024-32616.h5 | error exit | read 10 obj, 7 errors | read 10 obj, 6 errors | ok |
| cvefiles/cve-2024-32617.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
| cvefiles/cve-2024-32618.h5 | error exit | read 3 obj, 1 errors | read 3 obj, 1 errors | ok |
| cvefiles/cve-2024-32619.h5 | error exit | read 3 obj, 2 errors | read 3 obj, 2 errors | ok |
| cvefiles/cve-2024-32620.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
| cvefiles/cve-2024-32621.h5 | ok | read 3 obj, 1 errors | read 3 obj, 1 errors | ok |
| cvefiles/cve-2024-32622.h5 | ok | read 3 obj, 1 errors | read 3 obj, 1 errors | ok |
| cvefiles/cve-2024-32623.h5 | ok | read 6 obj | read 6 obj | ok |
| cvefiles/cve-2024-32624.h5 | error exit | read 6 obj, 1 errors | read 6 obj, 1 errors | ok |
| cvefiles/cve-2024-33873.h5 | error exit | read 4 obj, 1 errors | read 4 obj, 1 errors | ok |
| cvefiles/cve-2024-33874.h5 | ok | read 6 obj, 1 errors | read 6 obj, 1 errors | ok |
| cvefiles/cve-2024-33875.h5 | ok | read 2 obj | read 2 obj | ok |
| cvefiles/cve-2024-33876.h5 | ok | read 3 obj, 1 errors | read 3 obj, 1 errors | ok |
| cvefiles/cve-2024-33877.h5 | error exit | read 8 obj, 1 errors | read 8 obj, 1 errors | ok |
| cvefiles/cve-2025-2153.h5 | error exit | open error | read 1 obj, 1 errors | h5py-cannot-read |
| cvefiles/cve-2025-2308.h5 | error exit | read 25 obj, 1 errors | read 25 obj, 2 errors | our-error |
| cvefiles/cve-2025-2309.h5 | ok | read 6 obj, 1 errors | read 6 obj | ok |
| cvefiles/cve-2025-2310.h5 | error exit | read 24 obj, 8 errors | read 24 obj, 8 errors | ok |
| cvefiles/cve-2025-2912.h5 | error exit | open error | open error | h5py-cannot-read |
| cvefiles/cve-2025-2913.h5 | error exit | open error | open error | h5py-cannot-read |
| cvefiles/cve-2025-2914.h5 | error exit | open error | open error | h5py-cannot-read |
| cvefiles/cve-2025-2915.h5 | error exit | open error | open error | h5py-cannot-read |
| cvefiles/cve-2025-2923.h5 | error exit | open error | open error | h5py-cannot-read |
| cvefiles/cve-2025-2924.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
| cvefiles/cve-2025-2925.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
| cvefiles/cve-2025-2926.h5 | error exit | open error | open error | h5py-cannot-read |
| cvefiles/cve-2025-44904.h5 | error exit | read 25 obj, 1 errors | read 25 obj, 2 errors | our-error |
| cvefiles/cve-2025-44905.h5 | error exit | read 25 obj, 3 errors | read 25 obj, 3 errors | ok |
| cvefiles/cve-2025-6269-1.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
| cvefiles/cve-2025-6269-2.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
| cvefiles/cve-2025-6269-3.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
| cvefiles/cve-2025-6269-4.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
| cvefiles/cve-2025-6270-1.h5 | error exit | open error | open error | h5py-cannot-read |
| cvefiles/cve-2025-6270-2.h5 | error exit | open error | open error | h5py-cannot-read |
| cvefiles/cve-2025-6270-3.h5 | error exit | open error | open error | h5py-cannot-read |
| cvefiles/cve-2025-6516.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
| cvefiles/cve-2025-6750.h5 | error exit | open error | read 1 obj, 1 errors | h5py-cannot-read |
| cvefiles/cve-2025-6816.h5 | error exit | open error | open error | h5py-cannot-read |
| cvefiles/cve-2025-6817.h5 | error exit | open error | open error | h5py-cannot-read |
| cvefiles/cve-2025-6818.h5 | error exit | open error | open error | h5py-cannot-read |
| cvefiles/cve-2025-6856.h5 | error exit | open error | open error | h5py-cannot-read |
| cvefiles/cve-2025-6857.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
| cvefiles/cve-2025-6858.h5 | SIGSEGV | open error | open error | h5py-cannot-read |
| cvefiles/cve-2025-7067.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
| cvefiles/cve-2025-7068.h5 | error exit | open error | open error | h5py-cannot-read |
| cvefiles/cve-2025-7069.h5 | error exit | open error | open error | h5py-cannot-read |
| cvefiles/cve-2026-26200.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
| cvefiles/cve-2026-34734.h5 | error exit | read 2 obj, 1 errors | read 2 obj, 1 errors | ok |
| cvefiles/cve-2026-92627.h5 | error exit | read 2 obj, 1 errors | read 2 obj, 1 errors | ok |
| cvefiles/unknown-1.h5 | error exit | read 11 obj, 1 errors | read 11 obj, 1 errors | ok |
| fuzzerfiles/gh-4431-poc-03.h5 | error exit | read 1 obj | read 1 obj | ok |
| fuzzerfiles/gh-4432-poc-05.h5 | SIGSEGV | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
| fuzzerfiles/gh-4433-poc-08.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
| fuzzerfiles/gh-4434-poc-09.h5 | error exit | open error | read 1 obj, 1 errors | h5py-cannot-read |
| fuzzerfiles/gh-4435-poc-10.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
| fuzzerfiles/gh-4585.h5 | error exit | open error | open error | h5py-cannot-read |
| fuzzerfiles/gh_2649_flawed.h5 | error exit | read 9 obj, 1 errors | read 9 obj, 1 errors | ok |
| fuzzerfiles/gh_2649_plain_model.h5 | ok | read 10 obj | read 10 obj | ok |
</details>
## Known not-our-bug
- **h5py big-endian variable-length sequences.** h5py returns the elements of a VL sequence
whose base type is big-endian with the file's big-endian bytes but a native (little-endian)
numpy dtype, so the values it reports are byte-swapped garbage; `h5dump` prints the values
clawhdf5 reads. Reproducer: `h5py.vlen_dtype(np.dtype('>f4'))` dataset holding `[1.0, 2.0]`
reads back in h5py as `[4.6e-41, 9.0e-44]`. Affected here: `NCAS-CMS_pyfive/tests/data/attr_datatypes.hdf5`, `hdf5/tools/test/testfiles/tcomplex_be.h5`.
- **Non-IEEE floats and partial-precision integers (N-Bit).** libhdf5 converts a float whose
bit layout is not IEEE (e.g. `H5Tset_precision` for the N-Bit filter) or an integer with a
bit offset / reduced precision into the plain numpy type of the same size. The probe
compares such values as converted numbers, not raw file bytes (before 2026-09-25 it compared
raw bytes, which reported every N-Bit float dataset as a mismatch).
- **Types h5py widens.** Where h5py reads a type into a numpy type of a different size
(FP8 -> float16, bfloat16 -> float32, x87 long double -> float128) the values are not
compared (shape and presence still are): dataset file type size 1 -> numpy float16 (2) (15x), attr file type size 1 -> numpy float16 (2) (15x), dataset file type size 2 -> numpy float32 (4) (2x), dataset file type size 8 -> numpy float128 (16) (1x), dataset file type size 12 -> numpy float128 (16) (1x), attr file type size 2 -> numpy float32 (4) (1x), dataset file type size 2 -> numpy >f4 (4) (1x), attr file type size 2 -> numpy >f4 (4) (1x).
- **Corrupt data HDF5 2.0 reads through a bug.** clawhdf5 refuses these objects; h5py 3.16 /
HDF5 2.0 returns values for them that the file does not hold:
- `cve_hdf5/cvefiles/cve-2025-2308.h5` `/Scale_offset_long_long_data_le`: scale-offset codes run past the end of the chunk: HDF5 2.0 reads past its buffer; libhdf5's develop branch refuses the chunk ("Buffer too short").
- `cve_hdf5/cvefiles/cve-2025-44904.h5` `/Scale_offset_float_data_le`: unfiltered chunks of 38 and 37 bytes for 48-byte chunks: HDF5 2.0 fills the rest with whatever its buffer held; libhdf5's develop branch refuses them ("incorrect chunk size returned from index for unfiltered chunk").
- `hdf5/test/testfiles/bad_nbit_parms_walk.h5` `/Nbit_int_data_le`: an N-Bit parameter list one value short: HDF5 2.0 reads past the list; libhdf5's own test (`test_filter_bad_params`, test/dsets.c) now requires the read to fail.
- **References** are compared by presence only (`R`), not by target.
## Objects h5py fails on but clawhdf5 reads
- 19 x `OSError: Can't synchronously read data (no appropriate function for conversion path)`
- 1 x `TypeError: unhandled dtype kind M (dtype('…'))`
- 1 x `TypeError: No NumPy equivalent for TypeTimeID exists`
- 1 x `ValueError: Insufficient precision in available types to represent (N, N, N, N, N)`
## Reproduce
```sh
# needs: Rust, python3 with h5py numpy hdf5plugin (conformance/requirements.txt), h5dump (hdf5-tools), git
CLAWHDF5_PYTHON=/path/to/venv/bin/python conformance/run.sh
```
The corpus (about 450 MB of sparse checkouts) is cached in `conformance/.cache/`; results for
every file, both sides' raw JSON and stderr, are in `conformance/.cache/results/`.
`conformance/baseline.json` holds the ok files the nightly CI job (`.gitea/workflows/conformance.yml`)
must keep; `conformance/run.sh --update-baseline` rewrites it.
Cargo.toml
+16 -1
View File
@@ -16,13 +16,19 @@ members = [
"crates/clawhdf5-cli",
"crates/clawhdf5-napi",
"crates/clawhdf5-bench",
"crates/clawhdf5-tools",
"crates/clawhdf5-wasm",
"crates/clawhdf5-remote",
"crates/libaec-sys",
]
resolver = "2"
[workspace.package]
version = "2.4.0"
version = "2.7.0"
edition = "2024"
# Oldest toolchain that builds the whole workspace; CI checks it. wgpu (in
# clawhdf5-gpu) requires 1.92.
rust-version = "1.92"
license = "MIT"
repository = "https://git.redclaw.dev/quantumclaw/clawhdf5"
@@ -31,3 +37,12 @@ tempfile = "3"
criterion = { version = "0.5", features = ["html_reports"] }
half = "2.7"
serde = { version = "1", features = ["derive"] }
# The browser build of clawhdf5-wasm (examples/wasm-viewer/build.sh): size
# over speed, whole-program optimisation. Native profiles are unaffected.
[profile.wasm-release]
inherits = "release"
opt-level = "s"
lto = true
codegen-units = 1
panic = "abort"
README.md
+615 -147
View File
File diff suppressed because it is too large Load Diff
ROADMAP.md
+14 -8
View File
@@ -105,24 +105,30 @@
---
## Track 7: OpenClaw Integration
**Status:** 🟢 Complete
## Track 7: OpenClaw Integration — withdrawn (2026-09-25)
**Status:** ⚪ Withdrawn (the items below were library work; no OpenClaw integration shipped)
**Priority:** Critical (for adoption)
**Crates:** `clawhdf5-agent`, `clawhdf5-napi`
- [x] **7.1** Memory backend trait — MemoryBackend with search/get/write/ingest/export/stats
- [x] **7.2** Hybrid retrieval pipeline — ClawhdfBackend wires RRF → reranker → confidence rejection
- [x] **7.3** Markdown import/export — MarkdownParser + MarkdownExporter with line tracking + metadata
- [x] **7.4** memory_search tool — backed by full hybrid retrieval pipeline
- [x] **7.5** memory_get tool — get() with path + line range support
- [x] **7.4** `search()` — backed by the full hybrid retrieval pipeline (a Rust method; no OpenClaw tool was ever registered)
- [x] **7.5** `get()` — read back by path, with a line slice (not an OpenClaw tool either)
- [x] **7.6** Compaction integration — run_compaction() (decay + compact + WAL flush), run_consolidation() (hippocampal engine), tick_session(), flush_wal()
- [x] **7.7** Config surface — `memory.backend = "clawhdf5"` schema documented in docs/openclaw-config.md
- [x] **7.8** Documentation + migration guide — docs/migration-guide.md, docs/openclaw-integration.md (architecture, full API reference, code patterns)
- [ ] **7.7** ~~Config surface — `memory.backend = "clawhdf5"`~~ — never valid OpenClaw config; docs removed
- [ ] **7.8** ~~Documentation + migration guide~~ — removed: they described an integration that never worked
**Node.js bridge:** `clawhdf5-napi` (napi-rs) → `@redclaw/clawhdf5` npm package with full TypeScript types.
**Node.js bridge:** `clawhdf5-napi` (napi-rs) and a TypeScript wrapper in `packages/clawhdf5-node` exist but are unpublished, untested in CI and known to be broken (docs/known-issues.md).
---
> **Withdrawn.** None of this track produced a working OpenClaw integration: no
> plugin was built, the documented `memory.backend = "clawhdf5"` config was never
> valid in any OpenClaw release, and the Node package was never published. The
> Rust `ClawhdfBackend` remains as a library API. Not pursued for now; see
> [docs/openclaw.md](docs/openclaw.md) for what a plugin would need today.
## Track 8: Benchmarking & Validation
**Status:** 🟢 Complete
**Priority:** High
@@ -142,7 +148,7 @@
**Phase 1:** ~~Tracks 1, 2, 3 — core memory intelligence~~ 🟢 Complete
**Phase 2:** ~~Track 4 (temporal) + Track 5 (security)~~ 🟢 Complete
**Phase 3:** ~~Track 6 (multi-modal) + Track 7 (OpenClaw integration)~~ 🟢 Complete
**Phase 3:** ~~Track 6 (multi-modal)~~ 🟢 Complete; Track 7 (OpenClaw integration) withdrawn
**Phase 4:** ~~Track 8 (benchmarking + validation)~~ 🟢 Complete
All 8 tracks delivered. 1,650+ tests passing, zero clippy warnings.
conformance/.gitignore
+3
View File
@@ -0,0 +1,3 @@
/.cache/
# pin the probe's dependencies (the workspace lock is not committed)
!/probe/Cargo.lock
conformance/README.md
+39
View File
@@ -0,0 +1,39 @@
# Conformance sweep
Reads every HDF5 file of eight public corpora with clawhdf5 and with
h5py/libhdf5, compares the two readings object by object, and writes
[`CONFORMANCE.md`](../CONFORMANCE.md).
```sh
CLAWHDF5_PYTHON=/path/to/venv/bin/python conformance/run.sh # ~30 s once the corpus is cached
conformance/run.sh --update-baseline # after an intended change in results
```
Needs Rust, `git`, `h5dump` (Debian/Ubuntu `hdf5-tools`), `libaec` (for the
probe's `szip` feature; `libaec-dev`), and a Python with the packages in
`requirements.txt`. The first run downloads about 450 MB of sparse checkouts.
| file | role |
|---|---|
| `corpus.txt` | the corpora: git URL, pinned commit, swept root, sparse-checkout patterns |
| `fetch-corpus.sh` | shallow, sparse, blob-filtered checkout of each pinned commit into `.cache/src/` (gitignored); no-op when already there |
| `list_files.py` | which files are probed (HDF5/netCDF-4 extensions minus netCDF classic, plus the CVE reproducers) |
| `probe/` | the clawhdf5 side: a standalone crate (outside the workspace, so `cargo test --workspace` never builds it) that walks a file with `clawhdf5-format` and prints canonical JSON |
| `ref.py` | the h5py side: the same JSON from h5py |
| `run_one.sh` | runs both sides on one file (and `h5dump` on the CVE corpus) under a timeout and an address-space limit |
| `compare.py` | classifies each file (ok / our-error / mismatch / h5py-cannot-read / panic / hang / crash / oom) and groups root causes |
| `report.py` | writes `CONFORMANCE.md` |
| `check.py` | the gate: fails on any panic/hang/crash/oom, on an ok count below `baseline.json`, or on a baseline-ok file that is no longer ok |
| `baseline.json` | the ok files the gate holds the line on |
| `requirements.txt` | pinned h5py / numpy / hdf5plugin / netCDF4 |
Results for every file (both sides' JSON and stderr, `results.csv`,
`results.json`, `summary.md`) are left in `.cache/results/`.
The nightly job is `.gitea/workflows/conformance.yml`; it prints the report
into the job log.
The canonical value encoding both sides hash is documented at the top of
`probe/src/main.rs`. Values are compared as libhdf5 presents them: a float
with a non-IEEE bit layout (N-Bit) or an integer with a bit offset is compared
as the converted number, not as raw file bytes.
conformance/baseline.json
+648
View File
@@ -0,0 +1,648 @@
{
"comment": "conformance/run.sh fails if the ok count drops below `ok` or a file in `ok_files` stops being ok. Regenerate with `conformance/run.sh --update-baseline` after an intended change.",
"commit": "f37e7ae3263277319dba4bc39be5397194eb00c3",
"date": "2026-09-27 00:34 UTC",
"reference": "h5py 3.16.0 / HDF5 2.0.0",
"files": 697,
"ok": 600,
"counts": {
"h5py-cannot-read": 92,
"mismatch": 2,
"ok": 600,
"our-error": 3
},
"per_corpus": {
"NCAS-CMS_pyfive": {
"mismatch": 1,
"ok": 32
},
"cve_hdf5": {
"h5py-cannot-read": 32,
"ok": 113,
"our-error": 2
},
"h5py_data": {
"ok": 4
},
"hdf5": {
"h5py-cannot-read": 60,
"mismatch": 1,
"ok": 404,
"our-error": 1
},
"netcdf-c": {
"ok": 20
},
"netcdf4-python": {
"ok": 18
},
"usnistgov_h5wasm": {
"ok": 5
},
"xarray-data": {
"ok": 4
}
},
"ok_files": [
"NCAS-CMS_pyfive/tests/compact.hdf5",
"NCAS-CMS_pyfive/tests/data/btreev2.hdf5",
"NCAS-CMS_pyfive/tests/data/chunked.hdf5",
"NCAS-CMS_pyfive/tests/data/cmip_bad_eg.nc",
"NCAS-CMS_pyfive/tests/data/compressed.hdf5",
"NCAS-CMS_pyfive/tests/data/compressed_v1.hdf5",
"NCAS-CMS_pyfive/tests/data/dataset_datatypes.hdf5",
"NCAS-CMS_pyfive/tests/data/dataset_multidim.hdf5",
"NCAS-CMS_pyfive/tests/data/dim_scales.hdf5",
"NCAS-CMS_pyfive/tests/data/earliest.hdf5",
"NCAS-CMS_pyfive/tests/data/enum_h5variable.hdf5",
"NCAS-CMS_pyfive/tests/data/enum_variable.hdf5",
"NCAS-CMS_pyfive/tests/data/enum_variable.nc",
"NCAS-CMS_pyfive/tests/data/enums_from_netcdf.nc",
"NCAS-CMS_pyfive/tests/data/fillvalue_earliest.hdf5",
"NCAS-CMS_pyfive/tests/data/fillvalue_latest.hdf5",
"NCAS-CMS_pyfive/tests/data/filter_pipeline_v2.hdf5",
"NCAS-CMS_pyfive/tests/data/fletcher32.hdf5",
"NCAS-CMS_pyfive/tests/data/fractal_heap_no_mci_rlat.nc",
"NCAS-CMS_pyfive/tests/data/groups.hdf5",
"NCAS-CMS_pyfive/tests/data/h5netcdf_test.hdf5",
"NCAS-CMS_pyfive/tests/data/issue23_A.nc",
"NCAS-CMS_pyfive/tests/data/issue23_A_contiguous.nc",
"NCAS-CMS_pyfive/tests/data/issue23_B.nc",
"NCAS-CMS_pyfive/tests/data/latest.hdf5",
"NCAS-CMS_pyfive/tests/data/netcdf4_classic.nc",
"NCAS-CMS_pyfive/tests/data/new_style_groups.hdf5",
"NCAS-CMS_pyfive/tests/data/noy_AERmonZ_UKESM1-0-LL_piControl_r1i1p1f2_gnz_200001-200012.nc",
"NCAS-CMS_pyfive/tests/data/references.hdf5",
"NCAS-CMS_pyfive/tests/data/resizable.hdf5",
"NCAS-CMS_pyfive/tests/opaque_datetime.hdf5",
"NCAS-CMS_pyfive/tests/opaque_fixed.hdf5",
"cve_hdf5/cvefiles/cve-2016-4330.h5",
"cve_hdf5/cvefiles/cve-2016-4331.h5",
"cve_hdf5/cvefiles/cve-2016-4332-mtime-new.h5",
"cve_hdf5/cvefiles/cve-2016-4332-mtime.h5",
"cve_hdf5/cvefiles/cve-2016-4333.h5",
"cve_hdf5/cvefiles/cve-2017-17505.h5",
"cve_hdf5/cvefiles/cve-2017-17506.h5",
"cve_hdf5/cvefiles/cve-2017-17507.h5",
"cve_hdf5/cvefiles/cve-2017-17508.h5",
"cve_hdf5/cvefiles/cve-2017-17509.h5",
"cve_hdf5/cvefiles/cve-2018-11202.h5",
"cve_hdf5/cvefiles/cve-2018-11203.h5",
"cve_hdf5/cvefiles/cve-2018-11204.h5",
"cve_hdf5/cvefiles/cve-2018-11205.h5",
"cve_hdf5/cvefiles/cve-2018-11206-new.h5",
"cve_hdf5/cvefiles/cve-2018-11206-old.h5",
"cve_hdf5/cvefiles/cve-2018-11207.h5",
"cve_hdf5/cvefiles/cve-2018-13867.h5",
"cve_hdf5/cvefiles/cve-2018-13868.h5",
"cve_hdf5/cvefiles/cve-2018-13869.h5",
"cve_hdf5/cvefiles/cve-2018-13870.h5",
"cve_hdf5/cvefiles/cve-2018-13871.h5",
"cve_hdf5/cvefiles/cve-2018-13872.h5",
"cve_hdf5/cvefiles/cve-2018-13873.h5",
"cve_hdf5/cvefiles/cve-2018-13875.h5",
"cve_hdf5/cvefiles/cve-2018-14031.h5",
"cve_hdf5/cvefiles/cve-2018-14033.h5",
"cve_hdf5/cvefiles/cve-2018-14034.h5",
"cve_hdf5/cvefiles/cve-2018-14035.h5",
"cve_hdf5/cvefiles/cve-2018-14460.h5",
"cve_hdf5/cvefiles/cve-2018-15671.h5",
"cve_hdf5/cvefiles/cve-2018-15672.h5",
"cve_hdf5/cvefiles/cve-2018-16438.h5",
"cve_hdf5/cvefiles/cve-2018-17233.h5",
"cve_hdf5/cvefiles/cve-2018-17234.h5",
"cve_hdf5/cvefiles/cve-2018-17237.h5",
"cve_hdf5/cvefiles/cve-2018-17432.h5",
"cve_hdf5/cvefiles/cve-2018-17434.h5",
"cve_hdf5/cvefiles/cve-2018-17435.h5",
"cve_hdf5/cvefiles/cve-2018-17437.h5",
"cve_hdf5/cvefiles/cve-2018-17438",
"cve_hdf5/cvefiles/cve-2018-17439",
"cve_hdf5/cvefiles/cve-2019-8396.h5",
"cve_hdf5/cvefiles/cve-2019-8397.h5",
"cve_hdf5/cvefiles/cve-2019-8398.h5",
"cve_hdf5/cvefiles/cve-2019-9151.h5",
"cve_hdf5/cvefiles/cve-2019-9152.h5",
"cve_hdf5/cvefiles/cve-2020-10811.h5",
"cve_hdf5/cvefiles/cve-2020-18232.h5",
"cve_hdf5/cvefiles/cve-2020-18494.h5",
"cve_hdf5/cvefiles/cve-2021-36977.h5",
"cve_hdf5/cvefiles/cve-2021-37501.h5",
"cve_hdf5/cvefiles/cve-2021-45829.h5",
"cve_hdf5/cvefiles/cve-2021-45833.h5",
"cve_hdf5/cvefiles/cve-2021-46243.h5",
"cve_hdf5/cvefiles/cve-2021-46244.h5",
"cve_hdf5/cvefiles/cve-2024-29157.h5",
"cve_hdf5/cvefiles/cve-2024-29158.h5",
"cve_hdf5/cvefiles/cve-2024-29159.h5",
"cve_hdf5/cvefiles/cve-2024-29160.h5",
"cve_hdf5/cvefiles/cve-2024-29161.h5",
"cve_hdf5/cvefiles/cve-2024-29162.h5",
"cve_hdf5/cvefiles/cve-2024-29163.h5",
"cve_hdf5/cvefiles/cve-2024-29164.h5",
"cve_hdf5/cvefiles/cve-2024-29165.h5",
"cve_hdf5/cvefiles/cve-2024-29166.h5",
"cve_hdf5/cvefiles/cve-2024-32605.h5",
"cve_hdf5/cvefiles/cve-2024-32606.h5",
"cve_hdf5/cvefiles/cve-2024-32607-1.h5",
"cve_hdf5/cvefiles/cve-2024-32607-2.h5",
"cve_hdf5/cvefiles/cve-2024-32608.h5",
"cve_hdf5/cvefiles/cve-2024-32610.h5",
"cve_hdf5/cvefiles/cve-2024-32611.h5",
"cve_hdf5/cvefiles/cve-2024-32612.h5",
"cve_hdf5/cvefiles/cve-2024-32613.h5",
"cve_hdf5/cvefiles/cve-2024-32614.h5",
"cve_hdf5/cvefiles/cve-2024-32615.h5",
"cve_hdf5/cvefiles/cve-2024-32616.h5",
"cve_hdf5/cvefiles/cve-2024-32617.h5",
"cve_hdf5/cvefiles/cve-2024-32618.h5",
"cve_hdf5/cvefiles/cve-2024-32619.h5",
"cve_hdf5/cvefiles/cve-2024-32620.h5",
"cve_hdf5/cvefiles/cve-2024-32621.h5",
"cve_hdf5/cvefiles/cve-2024-32622.h5",
"cve_hdf5/cvefiles/cve-2024-32623.h5",
"cve_hdf5/cvefiles/cve-2024-32624.h5",
"cve_hdf5/cvefiles/cve-2024-33873.h5",
"cve_hdf5/cvefiles/cve-2024-33874.h5",
"cve_hdf5/cvefiles/cve-2024-33875.h5",
"cve_hdf5/cvefiles/cve-2024-33876.h5",
"cve_hdf5/cvefiles/cve-2024-33877.h5",
"cve_hdf5/cvefiles/cve-2025-2309.h5",
"cve_hdf5/cvefiles/cve-2025-2310.h5",
"cve_hdf5/cvefiles/cve-2025-2924.h5",
"cve_hdf5/cvefiles/cve-2025-2925.h5",
"cve_hdf5/cvefiles/cve-2025-44905.h5",
"cve_hdf5/cvefiles/cve-2025-6269-1.h5",
"cve_hdf5/cvefiles/cve-2025-6269-2.h5",
"cve_hdf5/cvefiles/cve-2025-6269-3.h5",
"cve_hdf5/cvefiles/cve-2025-6269-4.h5",
"cve_hdf5/cvefiles/cve-2025-6516.h5",
"cve_hdf5/cvefiles/cve-2025-6857.h5",
"cve_hdf5/cvefiles/cve-2025-7067.h5",
"cve_hdf5/cvefiles/cve-2026-26200.h5",
"cve_hdf5/cvefiles/cve-2026-34734.h5",
"cve_hdf5/cvefiles/cve-2026-92627.h5",
"cve_hdf5/cvefiles/unknown-1.h5",
"cve_hdf5/fuzzerfiles/gh-4431-poc-03.h5",
"cve_hdf5/fuzzerfiles/gh-4432-poc-05.h5",
"cve_hdf5/fuzzerfiles/gh-4433-poc-08.h5",
"cve_hdf5/fuzzerfiles/gh-4435-poc-10.h5",
"cve_hdf5/fuzzerfiles/gh_2649_flawed.h5",
"cve_hdf5/fuzzerfiles/gh_2649_plain_model.h5",
"h5py_data/compound-dtype-complex.h5",
"h5py_data/vlen_string_dset.h5",
"h5py_data/vlen_string_dset_utc.h5",
"h5py_data/vlen_string_s390x.h5",
"hdf5/HDF5Examples/C/H5FLT/tfiles/h5ex_d_bitgroom.h5",
"hdf5/HDF5Examples/C/H5FLT/tfiles/h5ex_d_blosc.h5",
"hdf5/HDF5Examples/C/H5FLT/tfiles/h5ex_d_blosc2.h5",
"hdf5/HDF5Examples/C/H5FLT/tfiles/h5ex_d_bshuf.h5",
"hdf5/HDF5Examples/C/H5FLT/tfiles/h5ex_d_bzip2.h5",
"hdf5/HDF5Examples/C/H5FLT/tfiles/h5ex_d_granularbr.h5",
"hdf5/HDF5Examples/C/H5FLT/tfiles/h5ex_d_jpeg.h5",
"hdf5/HDF5Examples/C/H5FLT/tfiles/h5ex_d_lz4.h5",
"hdf5/HDF5Examples/C/H5FLT/tfiles/h5ex_d_lzf.h5",
"hdf5/HDF5Examples/C/H5FLT/tfiles/h5ex_d_zfp.h5",
"hdf5/HDF5Examples/C/H5FLT/tfiles/h5ex_d_zstd.h5",
"hdf5/HDF5Examples/C/H5G/16/h5ex_g_iterate.h5",
"hdf5/HDF5Examples/C/H5G/16/h5ex_g_traverse.h5",
"hdf5/HDF5Examples/C/H5G/h5ex_g_iterate.h5",
"hdf5/HDF5Examples/C/H5G/h5ex_g_traverse.h5",
"hdf5/HDF5Examples/C/H5G/h5ex_g_visit.h5",
"hdf5/HDF5Examples/FORTRAN/H5G/h5ex_g_iterate.h5",
"hdf5/HDF5Examples/FORTRAN/H5G/h5ex_g_traverse.h5",
"hdf5/HDF5Examples/FORTRAN/H5G/h5ex_g_visit.h5",
"hdf5/HDF5Examples/JAVA/H5G/h5ex_g_iterate.h5",
"hdf5/HDF5Examples/JAVA/H5G/h5ex_g_visit.h5",
"hdf5/HDF5Examples/JAVA/compat/H5G/110/h5ex_g_iterate.h5",
"hdf5/HDF5Examples/JAVA/compat/H5G/110/h5ex_g_visit.h5",
"hdf5/HDF5Examples/JAVA/compat/H5G/h5ex_g_iterate.h5",
"hdf5/HDF5Examples/JAVA/compat/H5G/h5ex_g_visit.h5",
"hdf5/c++/test/th5s.h5",
"hdf5/hl/test/testfiles/test_ds_be.h5",
"hdf5/hl/test/testfiles/test_ds_be_new_ref-32bit.h5",
"hdf5/hl/test/testfiles/test_ds_be_new_ref.h5",
"hdf5/hl/test/testfiles/test_ds_le.h5",
"hdf5/hl/test/testfiles/test_ds_le_new_ref.h5",
"hdf5/hl/test/testfiles/test_ld.h5",
"hdf5/hl/test/testfiles/test_table_be.h5",
"hdf5/hl/test/testfiles/test_table_cray.h5",
"hdf5/hl/test/testfiles/test_table_le.h5",
"hdf5/test/testfiles/aggr.h5",
"hdf5/test/testfiles/bad_chunk_ndims.h5",
"hdf5/test/testfiles/bad_compound.h5",
"hdf5/test/testfiles/bad_offset.h5",
"hdf5/test/testfiles/be_data.h5",
"hdf5/test/testfiles/be_extlink1.h5",
"hdf5/test/testfiles/be_extlink2.h5",
"hdf5/test/testfiles/btree_idx_1_6.h5",
"hdf5/test/testfiles/btree_idx_1_8.h5",
"hdf5/test/testfiles/charsets.h5",
"hdf5/test/testfiles/corrupt_stab_msg.h5",
"hdf5/test/testfiles/deflate.h5",
"hdf5/test/testfiles/file_image_core_test.h5",
"hdf5/test/testfiles/filespace_1_6.h5",
"hdf5/test/testfiles/filespace_1_8.h5",
"hdf5/test/testfiles/fill18.h5",
"hdf5/test/testfiles/fill_old.h5",
"hdf5/test/testfiles/filter_error.h5",
"hdf5/test/testfiles/fsm_aggr_nopersist.h5",
"hdf5/test/testfiles/fsm_aggr_persist.h5",
"hdf5/test/testfiles/group_old.h5",
"hdf5/test/testfiles/h5fc_ext1_f.h5",
"hdf5/test/testfiles/h5fc_ext1_i.h5",
"hdf5/test/testfiles/h5fc_ext2_if.h5",
"hdf5/test/testfiles/h5fc_ext2_sf.h5",
"hdf5/test/testfiles/h5fc_ext3_isf.h5",
"hdf5/test/testfiles/h5fc_ext_none.h5",
"hdf5/test/testfiles/le_data.h5",
"hdf5/test/testfiles/le_extlink1.h5",
"hdf5/test/testfiles/le_extlink2.h5",
"hdf5/test/testfiles/memleak_H5O_dtype_decode_helper_H5Odtype.h5",
"hdf5/test/testfiles/mergemsg.h5",
"hdf5/test/testfiles/noencoder.h5",
"hdf5/test/testfiles/none.h5",
"hdf5/test/testfiles/paged_nopersist.h5",
"hdf5/test/testfiles/paged_persist.h5",
"hdf5/test/testfiles/specmetaread.h5",
"hdf5/test/testfiles/tarrold.h5",
"hdf5/test/testfiles/tbad_msg_count.h5",
"hdf5/test/testfiles/tbogus.h5",
"hdf5/test/testfiles/test_filters_be.h5",
"hdf5/test/testfiles/test_filters_le.h5",
"hdf5/test/testfiles/th5s.h5",
"hdf5/test/testfiles/tlayouto.h5",
"hdf5/test/testfiles/tmisc38a.h5",
"hdf5/test/testfiles/tmisc38b.h5",
"hdf5/test/testfiles/tmtimen.h5",
"hdf5/test/testfiles/tmtimeo.h5",
"hdf5/test/testfiles/tnullspace.h5",
"hdf5/test/testfiles/tsizeslheap.h5",
"hdf5/tools/test/testfiles/bigendian/tall.h5",
"hdf5/tools/test/testfiles/bigendian/tdset2.h5",
"hdf5/tools/test/testfiles/binfp64.h5",
"hdf5/tools/test/testfiles/binin16.h5",
"hdf5/tools/test/testfiles/binin32.h5",
"hdf5/tools/test/testfiles/binin8.h5",
"hdf5/tools/test/testfiles/binin8w.h5",
"hdf5/tools/test/testfiles/binuin16.h5",
"hdf5/tools/test/testfiles/binuin32.h5",
"hdf5/tools/test/testfiles/bounds_latest_latest.h5",
"hdf5/tools/test/testfiles/charsets.h5",
"hdf5/tools/test/testfiles/compounds_array_vlen1.h5",
"hdf5/tools/test/testfiles/compounds_array_vlen2.h5",
"hdf5/tools/test/testfiles/err_attr_dspace.h5",
"hdf5/tools/test/testfiles/file_space.h5",
"hdf5/tools/test/testfiles/filter_fail.h5",
"hdf5/tools/test/testfiles/h5clear_fsm_persist_equal.h5",
"hdf5/tools/test/testfiles/h5clear_fsm_persist_less.h5",
"hdf5/tools/test/testfiles/h5clear_fsm_persist_noclose.h5",
"hdf5/tools/test/testfiles/h5clear_fsm_persist_user_equal.h5",
"hdf5/tools/test/testfiles/h5clear_fsm_persist_user_less.h5",
"hdf5/tools/test/testfiles/h5clear_mdc_image.h5",
"hdf5/tools/test/testfiles/h5clear_sec2_v0.h5",
"hdf5/tools/test/testfiles/h5clear_sec2_v2.h5",
"hdf5/tools/test/testfiles/h5copy_extlinks_src.h5",
"hdf5/tools/test/testfiles/h5copy_extlinks_trg.h5",
"hdf5/tools/test/testfiles/h5copy_ref.h5",
"hdf5/tools/test/testfiles/h5copytst.h5",
"hdf5/tools/test/testfiles/h5copytst_new.h5",
"hdf5/tools/test/testfiles/h5diff_attr1.h5",
"hdf5/tools/test/testfiles/h5diff_attr2.h5",
"hdf5/tools/test/testfiles/h5diff_attr3.h5",
"hdf5/tools/test/testfiles/h5diff_attr_v_level1.h5",
"hdf5/tools/test/testfiles/h5diff_attr_v_level2.h5",
"hdf5/tools/test/testfiles/h5diff_basic1.h5",
"hdf5/tools/test/testfiles/h5diff_basic2.h5",
"hdf5/tools/test/testfiles/h5diff_comp_vl_strs.h5",
"hdf5/tools/test/testfiles/h5diff_danglelinks1.h5",
"hdf5/tools/test/testfiles/h5diff_danglelinks2.h5",
"hdf5/tools/test/testfiles/h5diff_dset1.h5",
"hdf5/tools/test/testfiles/h5diff_dset2.h5",
"hdf5/tools/test/testfiles/h5diff_dset3.h5",
"hdf5/tools/test/testfiles/h5diff_dset_zero_dim_size1.h5",
"hdf5/tools/test/testfiles/h5diff_dset_zero_dim_size2.h5",
"hdf5/tools/test/testfiles/h5diff_dtypes.h5",
"hdf5/tools/test/testfiles/h5diff_empty.h5",
"hdf5/tools/test/testfiles/h5diff_enum_invalid_values.h5",
"hdf5/tools/test/testfiles/h5diff_eps1.h5",
"hdf5/tools/test/testfiles/h5diff_eps2.h5",
"hdf5/tools/test/testfiles/h5diff_exclude1-1.h5",
"hdf5/tools/test/testfiles/h5diff_exclude1-2.h5",
"hdf5/tools/test/testfiles/h5diff_exclude2-1.h5",
"hdf5/tools/test/testfiles/h5diff_exclude2-2.h5",
"hdf5/tools/test/testfiles/h5diff_exclude3-1.h5",
"hdf5/tools/test/testfiles/h5diff_exclude3-2.h5",
"hdf5/tools/test/testfiles/h5diff_ext2softlink_src.h5",
"hdf5/tools/test/testfiles/h5diff_ext2softlink_trg.h5",
"hdf5/tools/test/testfiles/h5diff_extlink_src.h5",
"hdf5/tools/test/testfiles/h5diff_extlink_trg.h5",
"hdf5/tools/test/testfiles/h5diff_grp_recurse1.h5",
"hdf5/tools/test/testfiles/h5diff_grp_recurse2.h5",
"hdf5/tools/test/testfiles/h5diff_grp_recurse_ext1.h5",
"hdf5/tools/test/testfiles/h5diff_grp_recurse_ext2-1.h5",
"hdf5/tools/test/testfiles/h5diff_grp_recurse_ext2-2.h5",
"hdf5/tools/test/testfiles/h5diff_grp_recurse_ext2-3.h5",
"hdf5/tools/test/testfiles/h5diff_hyper1.h5",
"hdf5/tools/test/testfiles/h5diff_hyper2.h5",
"hdf5/tools/test/testfiles/h5diff_linked_softlink.h5",
"hdf5/tools/test/testfiles/h5diff_links.h5",
"hdf5/tools/test/testfiles/h5diff_onion_dset_1d.h5",
"hdf5/tools/test/testfiles/h5diff_onion_dset_ext.h5",
"hdf5/tools/test/testfiles/h5diff_onion_objs.h5",
"hdf5/tools/test/testfiles/h5diff_softlinks.h5",
"hdf5/tools/test/testfiles/h5diff_strings1.h5",
"hdf5/tools/test/testfiles/h5diff_strings2.h5",
"hdf5/tools/test/testfiles/h5diff_types.h5",
"hdf5/tools/test/testfiles/h5fc_edge_v3.h5",
"hdf5/tools/test/testfiles/h5fc_err_level.h5",
"hdf5/tools/test/testfiles/h5fc_ext1_f.h5",
"hdf5/tools/test/testfiles/h5fc_ext1_i.h5",
"hdf5/tools/test/testfiles/h5fc_ext1_s.h5",
"hdf5/tools/test/testfiles/h5fc_ext2_if.h5",
"hdf5/tools/test/testfiles/h5fc_ext2_is.h5",
"hdf5/tools/test/testfiles/h5fc_ext2_sf.h5",
"hdf5/tools/test/testfiles/h5fc_ext3_isf.h5",
"hdf5/tools/test/testfiles/h5fc_ext_none.h5",
"hdf5/tools/test/testfiles/h5fc_non_v3.h5",
"hdf5/tools/test/testfiles/h5repack_CVE-2018-14460.h5",
"hdf5/tools/test/testfiles/h5repack_CVE-2018-17432.h5",
"hdf5/tools/test/testfiles/h5repack_aggr.h5",
"hdf5/tools/test/testfiles/h5repack_attr.h5",
"hdf5/tools/test/testfiles/h5repack_attr_refs.h5",
"hdf5/tools/test/testfiles/h5repack_deflate.h5",
"hdf5/tools/test/testfiles/h5repack_early.h5",
"hdf5/tools/test/testfiles/h5repack_ext.h5",
"hdf5/tools/test/testfiles/h5repack_f32le.h5",
"hdf5/tools/test/testfiles/h5repack_f32le_ex.h5",
"hdf5/tools/test/testfiles/h5repack_fill.h5",
"hdf5/tools/test/testfiles/h5repack_filters.h5",
"hdf5/tools/test/testfiles/h5repack_fletcher.h5",
"hdf5/tools/test/testfiles/h5repack_fsm_aggr_nopersist.h5",
"hdf5/tools/test/testfiles/h5repack_fsm_aggr_persist.h5",
"hdf5/tools/test/testfiles/h5repack_hlink.h5",
"hdf5/tools/test/testfiles/h5repack_int32le_1d.h5",
"hdf5/tools/test/testfiles/h5repack_int32le_1d_ex.h5",
"hdf5/tools/test/testfiles/h5repack_int32le_2d.h5",
"hdf5/tools/test/testfiles/h5repack_int32le_2d_ex.h5",
"hdf5/tools/test/testfiles/h5repack_int32le_3d.h5",
"hdf5/tools/test/testfiles/h5repack_int32le_3d_ex.h5",
"hdf5/tools/test/testfiles/h5repack_layout.UD.h5",
"hdf5/tools/test/testfiles/h5repack_layout.h5",
"hdf5/tools/test/testfiles/h5repack_layout2.h5",
"hdf5/tools/test/testfiles/h5repack_layout3.h5",
"hdf5/tools/test/testfiles/h5repack_layouto.h5",
"hdf5/tools/test/testfiles/h5repack_named_dtypes.h5",
"hdf5/tools/test/testfiles/h5repack_nbit.h5",
"hdf5/tools/test/testfiles/h5repack_nested_8bit_enum.h5",
"hdf5/tools/test/testfiles/h5repack_nested_8bit_enum_deflated.h5",
"hdf5/tools/test/testfiles/h5repack_none.h5",
"hdf5/tools/test/testfiles/h5repack_objs.h5",
"hdf5/tools/test/testfiles/h5repack_paged_nopersist.h5",
"hdf5/tools/test/testfiles/h5repack_paged_persist.h5",
"hdf5/tools/test/testfiles/h5repack_refs.h5",
"hdf5/tools/test/testfiles/h5repack_shuffle.h5",
"hdf5/tools/test/testfiles/h5repack_soffset.h5",
"hdf5/tools/test/testfiles/h5repack_szip.h5",
"hdf5/tools/test/testfiles/h5repack_uint8be.h5",
"hdf5/tools/test/testfiles/h5repack_uint8be_ex.h5",
"hdf5/tools/test/testfiles/h5stat_err_old_fill.h5",
"hdf5/tools/test/testfiles/h5stat_err_old_layout.h5",
"hdf5/tools/test/testfiles/h5stat_err_refcount.h5",
"hdf5/tools/test/testfiles/h5stat_filters.h5",
"hdf5/tools/test/testfiles/h5stat_idx.h5",
"hdf5/tools/test/testfiles/h5stat_newgrat.h5",
"hdf5/tools/test/testfiles/h5stat_threshold.h5",
"hdf5/tools/test/testfiles/h5stat_tsohm.h5",
"hdf5/tools/test/testfiles/mod_h5clear_mdc_image.h5",
"hdf5/tools/test/testfiles/non_comparables1.h5",
"hdf5/tools/test/testfiles/non_comparables2.h5",
"hdf5/tools/test/testfiles/old_h5fc_ext1_f.h5",
"hdf5/tools/test/testfiles/old_h5fc_ext1_i.h5",
"hdf5/tools/test/testfiles/old_h5fc_ext1_s.h5",
"hdf5/tools/test/testfiles/old_h5fc_ext2_if.h5",
"hdf5/tools/test/testfiles/old_h5fc_ext2_is.h5",
"hdf5/tools/test/testfiles/old_h5fc_ext2_sf.h5",
"hdf5/tools/test/testfiles/old_h5fc_ext3_isf.h5",
"hdf5/tools/test/testfiles/old_h5fc_ext_none.h5",
"hdf5/tools/test/testfiles/packedbits.h5",
"hdf5/tools/test/testfiles/t128bit_float.h5",
"hdf5/tools/test/testfiles/tCVE-2021-37501_attr_decode.h5",
"hdf5/tools/test/testfiles/tCVE_2018_11206_fill_new.h5",
"hdf5/tools/test/testfiles/tCVE_2018_11206_fill_old.h5",
"hdf5/tools/test/testfiles/taindices.h5",
"hdf5/tools/test/testfiles/tall.h5",
"hdf5/tools/test/testfiles/tarray1.h5",
"hdf5/tools/test/testfiles/tarray1_big.h5",
"hdf5/tools/test/testfiles/tarray2.h5",
"hdf5/tools/test/testfiles/tarray3.h5",
"hdf5/tools/test/testfiles/tarray4.h5",
"hdf5/tools/test/testfiles/tarray5.h5",
"hdf5/tools/test/testfiles/tarray8.h5",
"hdf5/tools/test/testfiles/tattr.h5",
"hdf5/tools/test/testfiles/tattr2.h5",
"hdf5/tools/test/testfiles/tattr4_be.h5",
"hdf5/tools/test/testfiles/tattrintsize.h5",
"hdf5/tools/test/testfiles/tattrreg.h5",
"hdf5/tools/test/testfiles/tbfloat16.h5",
"hdf5/tools/test/testfiles/tbfloat16_be.h5",
"hdf5/tools/test/testfiles/tbigdims.h5",
"hdf5/tools/test/testfiles/tbinary.h5",
"hdf5/tools/test/testfiles/tbitnopaque.h5",
"hdf5/tools/test/testfiles/tchar.h5",
"hdf5/tools/test/testfiles/tcmpdattrintsize.h5",
"hdf5/tools/test/testfiles/tcmpdintarray.h5",
"hdf5/tools/test/testfiles/tcmpdints.h5",
"hdf5/tools/test/testfiles/tcmpdintsize.h5",
"hdf5/tools/test/testfiles/tcomplex.h5",
"hdf5/tools/test/testfiles/tcompound.h5",
"hdf5/tools/test/testfiles/tcompound_complex.h5",
"hdf5/tools/test/testfiles/tcompound_complex2.h5",
"hdf5/tools/test/testfiles/tdatareg.h5",
"hdf5/tools/test/testfiles/tdset.h5",
"hdf5/tools/test/testfiles/tdset2.h5",
"hdf5/tools/test/testfiles/tdset_idx.h5",
"hdf5/tools/test/testfiles/tempty.h5",
"hdf5/tools/test/testfiles/textlink.h5",
"hdf5/tools/test/testfiles/textlinkfar.h5",
"hdf5/tools/test/testfiles/textlinksrc.h5",
"hdf5/tools/test/testfiles/textlinktar.h5",
"hdf5/tools/test/testfiles/textpfe.h5",
"hdf5/tools/test/testfiles/tfcontents1.h5",
"hdf5/tools/test/testfiles/tfcontents2.h5",
"hdf5/tools/test/testfiles/tfilters.h5",
"hdf5/tools/test/testfiles/tfloat16.h5",
"hdf5/tools/test/testfiles/tfloat16_be.h5",
"hdf5/tools/test/testfiles/tfloat4.h5",
"hdf5/tools/test/testfiles/tfloat6.h5",
"hdf5/tools/test/testfiles/tfloat8.h5",
"hdf5/tools/test/testfiles/tfloatsattrs.h5",
"hdf5/tools/test/testfiles/tfpformat.h5",
"hdf5/tools/test/testfiles/tfvalues.h5",
"hdf5/tools/test/testfiles/tgroup.h5",
"hdf5/tools/test/testfiles/tgrp_comments.h5",
"hdf5/tools/test/testfiles/tgrpnullspace.h5",
"hdf5/tools/test/testfiles/thlink.h5",
"hdf5/tools/test/testfiles/thyperslab.h5",
"hdf5/tools/test/testfiles/tintascii.h5",
"hdf5/tools/test/testfiles/tints4dims.h5",
"hdf5/tools/test/testfiles/tintsattrs.h5",
"hdf5/tools/test/testfiles/tintsnodata.h5",
"hdf5/tools/test/testfiles/tlarge_objname.h5",
"hdf5/tools/test/testfiles/tldouble.h5",
"hdf5/tools/test/testfiles/tldouble_scalar.h5",
"hdf5/tools/test/testfiles/tlonglinks.h5",
"hdf5/tools/test/testfiles/tloop.h5",
"hdf5/tools/test/testfiles/tnamed_dtype_attr.h5",
"hdf5/tools/test/testfiles/tnestedcmpddt.h5",
"hdf5/tools/test/testfiles/tnestedcomp.h5",
"hdf5/tools/test/testfiles/tno-subset.h5",
"hdf5/tools/test/testfiles/tnullspace.h5",
"hdf5/tools/test/testfiles/torderattr.h5",
"hdf5/tools/test/testfiles/tordergr.h5",
"hdf5/tools/test/testfiles/trefer_attr.h5",
"hdf5/tools/test/testfiles/trefer_compat.h5",
"hdf5/tools/test/testfiles/trefer_ext1.h5",
"hdf5/tools/test/testfiles/trefer_ext2.h5",
"hdf5/tools/test/testfiles/trefer_grp.h5",
"hdf5/tools/test/testfiles/trefer_obj.h5",
"hdf5/tools/test/testfiles/trefer_obj_del.h5",
"hdf5/tools/test/testfiles/trefer_param.h5",
"hdf5/tools/test/testfiles/trefer_reg.h5",
"hdf5/tools/test/testfiles/trefer_reg_1d.h5",
"hdf5/tools/test/testfiles/tsaf.h5",
"hdf5/tools/test/testfiles/tscalarattrintsize.h5",
"hdf5/tools/test/testfiles/tscalarintattrsize.h5",
"hdf5/tools/test/testfiles/tscalarintsize.h5",
"hdf5/tools/test/testfiles/tscalarstring.h5",
"hdf5/tools/test/testfiles/tslink.h5",
"hdf5/tools/test/testfiles/tsoftlinks.h5",
"hdf5/tools/test/testfiles/tst_onion_dset_1d.h5",
"hdf5/tools/test/testfiles/tst_onion_dset_ext.h5",
"hdf5/tools/test/testfiles/tst_onion_objs.h5",
"hdf5/tools/test/testfiles/tstr.h5",
"hdf5/tools/test/testfiles/tstr2.h5",
"hdf5/tools/test/testfiles/tstr3.h5",
"hdf5/tools/test/testfiles/tudfilter.h5",
"hdf5/tools/test/testfiles/tudfilter2.h5",
"hdf5/tools/test/testfiles/tudlink.h5",
"hdf5/tools/test/testfiles/tvldtypes1.h5",
"hdf5/tools/test/testfiles/tvldtypes2.h5",
"hdf5/tools/test/testfiles/tvldtypes3.h5",
"hdf5/tools/test/testfiles/tvldtypes4.h5",
"hdf5/tools/test/testfiles/tvldtypes5.h5",
"hdf5/tools/test/testfiles/tvlenstr_array.h5",
"hdf5/tools/test/testfiles/tvlstr.h5",
"hdf5/tools/test/testfiles/tvms.h5",
"hdf5/tools/test/testfiles/twithub.h5",
"hdf5/tools/test/testfiles/twithub513.h5",
"hdf5/tools/test/testfiles/txtfp32.h5",
"hdf5/tools/test/testfiles/txtfp64.h5",
"hdf5/tools/test/testfiles/txtin16.h5",
"hdf5/tools/test/testfiles/txtin32.h5",
"hdf5/tools/test/testfiles/txtin8.h5",
"hdf5/tools/test/testfiles/txtstr.h5",
"hdf5/tools/test/testfiles/txtuin16.h5",
"hdf5/tools/test/testfiles/txtuin32.h5",
"hdf5/tools/test/testfiles/vds/1_a.h5",
"hdf5/tools/test/testfiles/vds/1_b.h5",
"hdf5/tools/test/testfiles/vds/1_c.h5",
"hdf5/tools/test/testfiles/vds/1_d.h5",
"hdf5/tools/test/testfiles/vds/1_e.h5",
"hdf5/tools/test/testfiles/vds/1_f.h5",
"hdf5/tools/test/testfiles/vds/1_vds.h5",
"hdf5/tools/test/testfiles/vds/2_a.h5",
"hdf5/tools/test/testfiles/vds/2_b.h5",
"hdf5/tools/test/testfiles/vds/2_c.h5",
"hdf5/tools/test/testfiles/vds/2_d.h5",
"hdf5/tools/test/testfiles/vds/2_e.h5",
"hdf5/tools/test/testfiles/vds/2_vds.h5",
"hdf5/tools/test/testfiles/vds/3_1_vds.h5",
"hdf5/tools/test/testfiles/vds/3_2_vds.h5",
"hdf5/tools/test/testfiles/vds/4_0.h5",
"hdf5/tools/test/testfiles/vds/4_1.h5",
"hdf5/tools/test/testfiles/vds/4_2.h5",
"hdf5/tools/test/testfiles/vds/4_vds.h5",
"hdf5/tools/test/testfiles/vds/5_a.h5",
"hdf5/tools/test/testfiles/vds/5_b.h5",
"hdf5/tools/test/testfiles/vds/5_c.h5",
"hdf5/tools/test/testfiles/vds/5_vds.h5",
"hdf5/tools/test/testfiles/vds/a.h5",
"hdf5/tools/test/testfiles/vds/b.h5",
"hdf5/tools/test/testfiles/vds/c.h5",
"hdf5/tools/test/testfiles/vds/d.h5",
"hdf5/tools/test/testfiles/vds/f-0.h5",
"hdf5/tools/test/testfiles/vds/f-3.h5",
"hdf5/tools/test/testfiles/vds/vds-eiger.h5",
"hdf5/tools/test/testfiles/vds/vds-percival-unlim-maxmin.h5",
"hdf5/tools/test/testfiles/xml/tbitfields.h5",
"hdf5/tools/test/testfiles/xml/tcompound2.h5",
"hdf5/tools/test/testfiles/xml/tdset2.h5",
"hdf5/tools/test/testfiles/xml/tenum.h5",
"hdf5/tools/test/testfiles/xml/test35.nc",
"hdf5/tools/test/testfiles/xml/tloop2.h5",
"hdf5/tools/test/testfiles/xml/tmany.h5",
"hdf5/tools/test/testfiles/xml/tname-amp.h5",
"hdf5/tools/test/testfiles/xml/tname-apos.h5",
"hdf5/tools/test/testfiles/xml/tname-gt.h5",
"hdf5/tools/test/testfiles/xml/tname-lt.h5",
"hdf5/tools/test/testfiles/xml/tname-quot.h5",
"hdf5/tools/test/testfiles/xml/tname-sp.h5",
"hdf5/tools/test/testfiles/xml/tnodata.h5",
"hdf5/tools/test/testfiles/xml/tobjref.h5",
"hdf5/tools/test/testfiles/xml/topaque.h5",
"hdf5/tools/test/testfiles/xml/tref-escapes-at.h5",
"hdf5/tools/test/testfiles/xml/tref-escapes.h5",
"hdf5/tools/test/testfiles/xml/tref.h5",
"hdf5/tools/test/testfiles/xml/tstring-at.h5",
"hdf5/tools/test/testfiles/xml/tstring.h5",
"hdf5/tools/test/testfiles/zerodim.h5",
"netcdf-c/h5_test/ref_tst_h_compounds.h5",
"netcdf-c/h5_test/ref_tst_h_compounds2.h5",
"netcdf-c/nc_test4/ref_hdf5_compat1.nc",
"netcdf-c/nc_test4/ref_hdf5_compat2.nc",
"netcdf-c/nc_test4/ref_hdf5_compat3.nc",
"netcdf-c/nc_test4/ref_szip.h5",
"netcdf-c/nc_test4/ref_tst_compounds.nc",
"netcdf-c/nc_test4/ref_tst_dims.nc",
"netcdf-c/nc_test4/ref_tst_interops4.nc",
"netcdf-c/nc_test4/ref_tst_xplatform2_1.nc",
"netcdf-c/nc_test4/ref_tst_xplatform2_2.nc",
"netcdf-c/nc_test4/tdset.h5",
"netcdf-c/ncdump/ref_nc_test_netcdf4_4_0.nc",
"netcdf-c/ncdump/ref_no_ncproperty.nc",
"netcdf-c/ncdump/ref_provenance_v1.nc",
"netcdf-c/ncdump/ref_test_corrupt_magic.nc",
"netcdf-c/ncdump/ref_tst_compounds2.nc",
"netcdf-c/ncdump/ref_tst_compounds3.nc",
"netcdf-c/ncdump/ref_tst_compounds4.nc",
"netcdf-c/ncdump/ref_tst_irish_rover.nc",
"netcdf4-python/examples/data/prmsl.2000.nc",
"netcdf4-python/examples/data/prmsl.2001.nc",
"netcdf4-python/examples/data/prmsl.2002.nc",
"netcdf4-python/examples/data/prmsl.2003.nc",
"netcdf4-python/examples/data/prmsl.2004.nc",
"netcdf4-python/examples/data/prmsl.2005.nc",
"netcdf4-python/examples/data/prmsl.2006.nc",
"netcdf4-python/examples/data/prmsl.2007.nc",
"netcdf4-python/examples/data/prmsl.2008.nc",
"netcdf4-python/examples/data/prmsl.2009.nc",
"netcdf4-python/examples/data/prmsl.2010.nc",
"netcdf4-python/examples/data/prmsl.2011.nc",
"netcdf4-python/examples/data/rtofs_glo_3dz_f006_6hrly_reg3.nc",
"netcdf4-python/test/20171025_2056.Cloud_Top_Height.nc",
"netcdf4-python/test/issue1152.nc",
"netcdf4-python/test/issue671.nc",
"netcdf4-python/test/issue672.nc",
"netcdf4-python/test/test_gold.nc",
"usnistgov_h5wasm/test/array.h5",
"usnistgov_h5wasm/test/compressed.h5",
"usnistgov_h5wasm/test/empty.h5",
"usnistgov_h5wasm/test/float16.h5",
"usnistgov_h5wasm/test/vlen.h5",
"xarray-data/ROMS_example.nc",
"xarray-data/basin_mask.nc",
"xarray-data/imerghh_730.hdf5",
"xarray-data/precipitation.nc4"
]
}
conformance/check.py
Executable
+88
View File
@@ -0,0 +1,88 @@
#!/usr/bin/env python3
"""check.py <results_dir> <baseline.json> [--update]
The conformance gate. Fails (exit 1) when
* clawhdf5 panicked, hung, crashed or ran out of memory on any file, or
* the ok count fell below the baseline's, or
* a file the baseline lists as ok is no longer ok (even if another file
became ok and the total held).
New ok files are reported so the baseline can be raised (--update rewrites it
from the results).
"""
import json
import os
import sys
FATAL = ("panic", "hang", "crash", "oom")
def main():
args = [a for a in sys.argv[1:] if not a.startswith("--")]
update = "--update" in sys.argv
res_dir, base_path = args
res = json.load(open(os.path.join(res_dir, "results.json")))
rows = res["rows"]
counts = {}
per_corpus = {}
for r in rows:
counts[r["class"]] = counts.get(r["class"], 0) + 1
pc = per_corpus.setdefault(r["corpus"], {})
pc[r["class"]] = pc.get(r["class"], 0) + 1
ok_files = sorted(r["file"] for r in rows if r["class"] == "ok")
if update:
meta = {}
mp = os.path.join(res_dir, "report-meta.json")
if os.path.exists(mp):
meta = json.load(open(mp))
base = {
"comment": "conformance/run.sh fails if the ok count drops below `ok` or a file in `ok_files` stops being ok. "
"Regenerate with `conformance/run.sh --update-baseline` after an intended change.",
"commit": meta.get("commit", ""),
"date": meta.get("date", ""),
"reference": meta.get("reference", ""),
"files": len(rows),
"ok": len(ok_files),
"counts": dict(sorted(counts.items())),
"per_corpus": {k: dict(sorted(v.items())) for k, v in sorted(per_corpus.items())},
"ok_files": ok_files,
}
with open(base_path, "w") as fh:
json.dump(base, fh, indent=1)
fh.write("\n")
print(f"baseline updated: {len(ok_files)} ok of {len(rows)} files -> {base_path}")
return 0
base = json.load(open(base_path))
failures = []
fatal = [r for r in rows if r["class"] in FATAL]
for r in fatal:
failures.append(f"{r['class']}: {r['file']}: {r['ours_detail'][:200]}")
if len(ok_files) < base["ok"]:
failures.append(f"ok count dropped: {len(ok_files)} < baseline {base['ok']}")
now_ok = set(ok_files)
by_file = {r["file"]: r for r in rows}
for f in base["ok_files"]:
if f not in now_ok:
r = by_file.get(f)
why = f"now {r['class']}: {(r['ours_detail'] or r['first_issue'])[:200]}" if r else "no longer in the corpus"
failures.append(f"regressed: {f}: {why}")
gained = sorted(now_ok - set(base["ok_files"]))
print(f"conformance: {len(ok_files)} ok of {len(rows)} files (baseline {base['ok']} of {base['files']}); "
+ ", ".join(f"{k} {v}" for k, v in sorted(counts.items())))
if gained:
print(f"{len(gained)} file(s) newly ok — raise the baseline with `conformance/run.sh --update-baseline`:")
for f in gained:
print(f" + {f}")
if failures:
print(f"CONFORMANCE GATE FAILED ({len(failures)}):")
for f in failures:
print(f" - {f}")
return 1
print("conformance gate passed")
return 0
if __name__ == "__main__":
sys.exit(main())
conformance/compare.py
Executable
+295
View File
@@ -0,0 +1,295 @@
#!/usr/bin/env python3
"""compare.py <results_dir>: classify each file and group failures by root cause.
Writes <results_dir>/results.csv, results.json and summary.md.
File classes (first match wins):
hang, oom, crash, panic ours: timeout / allocation failure / signal / any panic (caught or not)
h5py-cannot-read libhdf5/h5py failed to open the file (or crashed/hung)
our-error we fail to open, list, or read something h5py reads
mismatch we read something with different shape/values, or a different object set
ok
"""
import collections
import csv
import json
import os
import re
import sys
R = sys.argv[1]
RUNS = os.path.join(R, "runs")
def load(d, name):
rc_p = os.path.join(d, name + ".rc")
if not os.path.exists(rc_p):
return None
rc = int(open(rc_p).read().strip() or -1)
err = open(os.path.join(d, name + ".err"), errors="replace").read()
js = None
try:
js = json.load(open(os.path.join(d, name + ".json")))
except Exception: # noqa: BLE001
pass
return {"rc": rc, "err": err, "json": js}
def proc_status(p):
"""-> (status, detail)"""
if p is None:
return "missing", ""
rc, err = p["rc"], p["err"]
first_panic = next((ln for ln in err.splitlines() if ln.startswith("PANIC:") or "panicked at" in ln), "")
if rc == 0 and p["json"] is not None:
return "ok", ""
if rc == 137 or rc == 124:
return "hang", f"timeout ({os.environ.get('TMO', '20')} s)"
if "memory allocation of" in err or "MemoryError" in err or "std::bad_alloc" in err:
m = re.search(r"memory allocation of \d+ bytes failed", err)
return "oom", m.group(0) if m else "allocation failure"
if "overflowed its stack" in err:
return "crash", "stack overflow"
if rc == 101:
return "panic", first_panic or (err.strip().splitlines() or [""])[-1]
if rc in (134, 139, 136, 135, 132) or rc > 128:
sig = {134: "SIGABRT", 139: "SIGSEGV", 136: "SIGFPE", 135: "SIGBUS", 132: "SIGILL"}.get(rc, f"signal {rc - 128}")
tail = [ln for ln in err.strip().splitlines() if ln.strip()][-1:]
return "crash", f"{sig}: {tail[0][:200] if tail else ''}"
tail = [ln for ln in err.strip().splitlines() if ln.strip()][-1:]
return "crash", f"rc={rc}: {tail[0][:200] if tail else ''}"
def norm(msg):
m = msg.split("\n")[0]
m = re.sub(r"0x[0-9a-fA-F]+", "X", m)
m = re.sub(r'"[^"]*"', '"…"', m)
m = re.sub(r"'[^']*'", "'…'", m)
m = re.sub(r"\d+", "N", m)
return m[:160]
def panic_head(msg):
"""First line + first clawhdf5 frame of a PANIC record."""
lines = msg.split("\n")
frame = next((ln.strip() for ln in lines[1:] if "clawhdf5_format" in ln), "")
return lines[0][:300], frame[:300]
def eq_shape(a, b):
return a == b
rows = []
issues_by_file = {}
root_causes = collections.defaultdict(lambda: {"files": set(), "count": 0, "examples": []})
mismatch_causes = collections.defaultdict(lambda: {"files": set(), "count": 0, "examples": []})
panics = []
ref_only_errors = collections.Counter()
incomparable = collections.Counter()
def add(bucket, key, file, example):
b = bucket[key]
b["count"] += 1
if file not in b["files"] and len(b["examples"]) < 6:
b["examples"].append(example)
b["files"].add(file)
files = [ln.strip() for ln in open(os.path.join(R, "files.txt")) if ln.strip()]
for rel in files:
d = os.path.join(RUNS, rel.replace("/", "__"))
corpus = rel.split("/")[0]
ours, ref = load(d, "ours"), load(d, "ref")
h5dump = load(d, "h5dump")
os_, od = proc_status(ours)
rs, rd = proc_status(ref)
oj = ours["json"] if ours else None
rj = ref["json"] if ref else None
issues = [] # (kind, detail)
caught_panics = []
def scan_err(path, what, msg):
if msg.startswith("PANIC:"):
caught_panics.append((path, what, msg))
if oj:
for o in oj.get("objects", []):
for k in ("error", "attrs_error", "list_error"):
if k in o:
scan_err(o["path"], k, o[k])
for an, av in (o.get("attrs") or {}).items():
if "error" in av:
scan_err(o["path"], f"attr {an}", av["error"])
if oj.get("open_error", "").startswith("PANIC:"):
caught_panics.append(("<open>", "open", oj["open_error"]))
ref_open_fail = rs != "ok" or (rj is not None and "open_error" in rj)
ours_open_err = oj.get("open_error") if oj else None
n_obj = n_ok = 0
if os_ == "ok" and rj and not ref_open_fail and not ours_open_err:
ro = {x["path"]: x for x in rj.get("objects", [])}
oo = {x["path"]: x for x in oj.get("objects", [])}
our_list_errors = [x for x in oo.values() if "list_error" in x]
for p in sorted(set(ro) | set(oo)):
a, b = ro.get(p), oo.get(p)
n_obj += 1
if a is None:
issues.append(("mismatch", f"extra object {p} (kind={b.get('kind')})", "extra-object", b))
continue
if b is None:
if our_list_errors:
continue # accounted for by the list_error
issues.append(("mismatch", f"missing object {p} (kind={a.get('kind')})", "missing-object", a))
continue
ok = True
if a.get("kind") != b.get("kind") and "error" not in b and "error" not in a:
issues.append(("mismatch", f"{p}: kind {a.get('kind')} vs ours {b.get('kind')}", "kind", b))
ok = False
# h5py could not open the object at all: it read none of its
# attributes or links, so there is nothing to compare ours with
# (the object's own error is compared above and below).
ref_unopened = a.get("kind") == "unknown" and "error" in a
for k in ("error", "list_error", "attrs_error"):
if ref_unopened and k != "error":
continue
if k in b and k not in a:
issues.append(("our-error", f"{p}: {k}: {b[k]}", b[k], b))
ok = False
elif k in a and k not in b and k == "error":
ref_only_errors[norm(a[k])] += 1
if a.get("kind") == "dataset" and "error" not in a and "error" not in b:
if "skipped" in a or "skipped" in b:
pass
elif a.get("converted"):
incomparable[f"dataset {a['converted']}"] += 1
elif a.get("shape") != b.get("shape"):
issues.append(("mismatch", f"{p}: shape {a.get('shape')} vs ours {b.get('shape')}", "shape", b))
ok = False
elif a.get("hash") != b.get("hash"):
issues.append(("mismatch", f"{p}: values differ (h5py {a.get('dtype')} vs ours {b.get('dtype')})", "values", b | {"ref_head": a.get("head"), "ref_dtype": a.get("dtype")}))
ok = False
ra, oa = a.get("attrs") or {}, b.get("attrs") or {}
if "attrs_error" not in b and "attrs_error" not in a and not ref_unopened:
for an in sorted(set(ra) | set(oa)):
x, y = ra.get(an), oa.get(an)
if x is None:
issues.append(("mismatch", f"{p}@{an}: extra attribute", "extra-attr", y or {}))
elif y is None:
issues.append(("mismatch", f"{p}@{an}: missing attribute", "missing-attr", x))
elif "error" in y and "error" not in x:
issues.append(("our-error", f"{p}@{an}: {y['error']}", y["error"], y))
elif "error" in x:
continue
elif x.get("converted"):
incomparable[f"attr {x['converted']}"] += 1
elif x.get("shape") != y.get("shape"):
issues.append(("mismatch", f"{p}@{an}: attr shape {x.get('shape')} vs ours {y.get('shape')}", "attr-shape", y | {"ref_dtype": x.get("dtype")}))
elif x.get("hash") != y.get("hash"):
issues.append(("mismatch", f"{p}@{an}: attr values differ (h5py {x.get('dtype')} vs ours {y.get('dtype')})", "attr-values", y | {"ref_head": x.get("head"), "ref_dtype": x.get("dtype")}))
if ok:
n_ok += 1
# classify
if os_ in ("hang", "oom", "crash", "panic"):
cls = os_
elif caught_panics:
cls = "panic"
elif ref_open_fail:
cls = "h5py-cannot-read"
elif ours_open_err:
cls = "our-error"
issues.append(("our-error", f"open: {ours_open_err}", ours_open_err, {}))
elif any(i[0] == "our-error" for i in issues):
cls = "our-error"
elif issues:
cls = "mismatch"
else:
cls = "ok"
if os_ in ("hang", "oom", "crash", "panic") or caught_panics:
panics.append({
"file": rel, "class": cls, "detail": od,
"stderr": (ours["err"] if ours else "")[:3000],
"caught": [(p, w, m[:2500]) for p, w, m in caught_panics[:3]],
"n_caught": len(caught_panics),
})
for kind, detail, key, rec in issues:
if kind == "our-error":
add(root_causes, norm(key), rel, detail[:300])
else:
if key in ("values", "attr-values", "shape", "attr-shape"):
mk = f"{key}: ours={rec.get('dtype')} h5py={rec.get('ref_dtype')} layout={rec.get('layout','-')} filters={rec.get('filters','-')}"
else:
mk = key
add(mismatch_causes, mk, rel, detail[:300] + (f" | ref_head={rec.get('ref_head')} our_head={rec.get('head')}" if rec.get("ref_head") else ""))
ref_detail = rd if rs != "ok" else ((rj or {}).get("open_error") or "")
h5d = ""
if h5dump:
rc = h5dump["rc"]
h5d = {0: "ok", 1: "error", 137: "hang", 124: "hang", 134: "SIGABRT", 139: "SIGSEGV", 136: "SIGFPE", 135: "SIGBUS"}.get(rc, f"rc={rc}")
if "memory allocation" in h5dump["err"] or "Cannot allocate" in h5dump["err"]:
h5d += "(oom)"
rows.append({
"file": rel, "corpus": corpus, "class": cls,
"ours": os_ if os_ != "ok" else ("open-error" if ours_open_err else ("panic" if caught_panics else "ok")),
"ours_detail": (od or ours_open_err or (caught_panics[0][2].split("\n")[0] if caught_panics else ""))[:300],
"ref": rs if rs != "ok" else ("open-error" if (rj or {}).get("open_error") else "ok"),
"ref_detail": ref_detail[:300],
"h5dump_1_14_6": h5d,
"h5dump_detail": ([ln for ln in h5dump["err"].splitlines() if ln.strip()][-1:] or [""])[0][:200] if h5dump else "",
"objects": n_obj, "objects_ok": n_ok,
"issues": len(issues), "first_issue": issues[0][1][:300] if issues else "",
"superblock": (oj or {}).get("superblock_version", ""),
})
# the first issues of each file, for report.py's known-cause matching
issues_by_file[rel] = [
{"kind": k, "key": key, "detail": det[:300], "ours_dtype": rec.get("dtype"), "ref_dtype": rec.get("ref_dtype")}
for k, det, key, rec in issues[:50]
]
with open(os.path.join(R, "results.csv"), "w", newline="") as fh:
w = csv.DictWriter(fh, fieldnames=list(rows[0].keys()))
w.writeheader()
w.writerows(rows)
def ser(b):
return {k: {"files": len(v["files"]), "count": v["count"], "examples": v["examples"], "file_list": sorted(v["files"])} for k, v in sorted(b.items(), key=lambda kv: -len(kv[1]["files"]))}
json.dump({"rows": rows, "issues": issues_by_file, "root_causes": ser(root_causes), "mismatch_causes": ser(mismatch_causes),
"panics": panics, "incomparable": incomparable.most_common(), "ref_only_errors": ref_only_errors.most_common()},
open(os.path.join(R, "results.json"), "w"), indent=1)
classes = ["ok", "our-error", "mismatch", "h5py-cannot-read", "hang", "panic", "crash", "oom"]
by_corpus = collections.defaultdict(collections.Counter)
for r in rows:
by_corpus[r["corpus"]][r["class"]] += 1
by_corpus["ALL"][r["class"]] += 1
lines = ["# Conformance sweep summary", "", "| corpus | files | " + " | ".join(classes) + " |", "|---" * (len(classes) + 2) + "|"]
for c in sorted(by_corpus, key=lambda k: (k == "ALL", k)):
cnt = by_corpus[c]
lines.append(f"| {c} | {sum(cnt.values())} | " + " | ".join(str(cnt.get(k, 0)) for k in classes) + " |")
lines += ["", "## Panics / hangs / crashes / OOM", ""]
for p in panics:
lines.append(f"- **{p['file']}** [{p['class']}] {p['detail']}")
for path, what, m in p["caught"][:1]:
lines.append(" ```\n " + f"{path} ({what}): " + m.replace("\n", "\n ")[:1500] + "\n ```")
if not p["caught"] and p["stderr"]:
lines.append(" ```\n " + p["stderr"].strip()[:1500].replace("\n", "\n ") + "\n ```")
lines += ["", "## Our-error root causes (files affected)", ""]
for k, v in ser(root_causes).items():
lines.append(f"- [{v['files']} files, {v['count']} objs] `{k}`")
for ex in v["examples"][:3]:
lines.append(f" - {ex}")
lines += ["", "## Mismatch root causes", ""]
for k, v in ser(mismatch_causes).items():
lines.append(f"- [{v['files']} files, {v['count']} objs] `{k}`")
for ex in v["examples"][:3]:
lines.append(f" - {ex}")
lines += ["", "## Objects h5py fails on but we read (top)", ""]
for k, n in ref_only_errors.most_common(15):
lines.append(f"- {n} x `{k}`")
open(os.path.join(R, "summary.md"), "w").write("\n".join(lines) + "\n")
print("\n".join(lines[:4 + len(by_corpus)]))
conformance/corpus.txt
+19
View File
@@ -0,0 +1,19 @@
# Conformance corpora, pinned by commit. fetch-corpus.sh reads this file.
#
# name git-url commit root [sparse-checkout patterns...]
#
# `root` is the directory inside the checkout that is swept ("." = all of it).
# Patterns are git non-cone sparse-checkout patterns; none = whole repository.
# Every file under <root> with an HDF5/netCDF-4 extension is probed; for
# cve_hdf5 the extension-less files in cvefiles/ and fuzzerfiles/ are too.
# Licences: each corpus keeps its upstream licence; nothing here is committed
# to this repository — the files are downloaded into the gitignored cache.
hdf5 https://github.com/HDFGroup/hdf5.git a3cf1ea82cc7a66e50029a688121e1b105a7ce88 . *.h5 *.he5 *.nc *.hdf5 *.h5f
cve_hdf5 https://github.com/HDFGroup/cve_hdf5.git 3fd1f5ae3869e01b8ae02b41d7108de7ffb1a374 .
netcdf-c https://github.com/Unidata/netcdf-c.git beb7b9585273c1548386231a59b809d906359033 . /nc_test4/*.nc /ncdump/*.nc /nc_test4/*.h5 /ncdump/*.h5 /h5_test/*.h5 /hdf5_test/*.h5
NCAS-CMS_pyfive https://github.com/NCAS-CMS/pyfive.git 8cf07b8749133f41c5e30b8a4c604486f687fe74 . *.h5 *.hdf5 *.hdf *.nc *.he5
usnistgov_h5wasm https://github.com/usnistgov/h5wasm.git 02f6336527d2812783fcedabfbf42127ec8d06d2 . *.h5 *.hdf5 *.hdf *.nc *.he5
netcdf4-python https://github.com/Unidata/netcdf4-python.git 6e67576d39aef8091fb20bd767b4f1a52ddc1bec . *.nc *.h5
xarray-data https://github.com/pydata/xarray-data.git a35297e9da2cc99c811014f0c8a4297345a5c28d . /basin_mask.nc /precipitation.nc4 /imerghh_730.hdf5 /eraint_uvz.nc /ROMS_example.nc /tiny.nc
# h5py 3.16.0 (tag 3.16.0), its test data files.
h5py_data https://github.com/h5py/h5py.git b2f0347c4200333acd89b43733f1caa0c115162f h5py/tests/data_files /h5py/tests/data_files/*
conformance/fetch-corpus.sh
Executable
+39
View File
@@ -0,0 +1,39 @@
#!/usr/bin/env bash
# fetch-corpus.sh [cache_dir]
#
# Download the corpora pinned in conformance/corpus.txt into the (gitignored)
# cache: <cache>/src/<name> is a shallow, sparse, blob-filtered checkout of the
# pinned commit and <cache>/corpus/<name> links to the swept root inside it.
# A corpus already checked out at its pinned commit is left alone, so a second
# run costs nothing and needs no network.
set -euo pipefail
HERE="$(cd "$(dirname "$0")" && pwd)"
CACHE="${1:-${CONFORMANCE_CACHE:-$HERE/.cache}}"
mkdir -p "$CACHE/src" "$CACHE/corpus"
CACHE="$(cd "$CACHE" && pwd)"
retry() { local i; for i in 1 2 3 4; do "$@" && return 0; sleep $((i * 5)); done; return 1; }
grep -v '^[[:space:]]*\(#\|$\)' "$HERE/corpus.txt" | while read -r name url commit root patterns; do
src="$CACHE/src/$name"
if [ -d "$src/.git" ] && [ "$(git -C "$src" rev-parse HEAD 2>/dev/null)" = "$commit" ]; then
echo "cached $name @ ${commit:0:12}"
else
echo "fetching $name @ ${commit:0:12} from $url"
rm -rf "$src"
git init -q "$src"
git -C "$src" remote add origin "$url"
git -C "$src" config advice.detachedHead false
if [ -n "$patterns" ]; then
git -C "$src" config core.sparseCheckout true
# no-cone patterns (globs); `set -f` keeps the shell from expanding them
(set -f; printf '%s\n' $patterns) > "$src/.git/info/sparse-checkout"
fi
retry git -C "$src" fetch -q --depth 1 --filter=blob:none origin "$commit"
retry git -C "$src" checkout -q FETCH_HEAD
got="$(git -C "$src" rev-parse HEAD)"
[ "$got" = "$commit" ] || { echo "error: $name checked out $got, expected $commit" >&2; exit 1; }
fi
ln -sfn "$src/$root" "$CACHE/corpus/$name"
done
echo "corpus ready in $CACHE/corpus"
conformance/list_files.py
+48
View File
@@ -0,0 +1,48 @@
#!/usr/bin/env python3
"""list_files.py <corpus_dir>: print the files the sweep probes, one per line,
as <corpus>/<path> in byte order.
* every file named *.h5 *.hdf5 *.he5 *.nc *.nc4 *.hdf *.h5f in each corpus,
except netCDF classic / 64-bit-offset / CDF5 files (magic "CDF"): they are
not HDF5, so neither side can read them and they say nothing;
* plus, for cve_hdf5, every file in cvefiles/ and fuzzerfiles/ except
.md/.c sources — the reproducers are mostly extension-less, and they are
kept whatever their bytes look like (that is their point).
"""
import os
import sys
EXTS = (".h5", ".hdf5", ".he5", ".nc", ".nc4", ".hdf", ".h5f")
def walk(top):
for dirpath, dirnames, filenames in os.walk(top):
dirnames[:] = [d for d in dirnames if d != ".git"]
for fn in filenames:
p = os.path.join(dirpath, fn)
if os.path.isfile(p) and not os.path.islink(p):
yield os.path.relpath(p, top)
def main(root):
out = set()
for corpus in sorted(os.listdir(root)):
top = os.path.join(root, corpus)
if not os.path.isdir(top):
continue
for rel in walk(top):
path = os.path.join(top, rel)
if rel.lower().endswith(EXTS):
with open(path, "rb") as fh:
if fh.read(3) == b"CDF":
continue
out.add(f"{corpus}/{rel}")
elif corpus == "cve_hdf5" and rel.split(os.sep)[0] in ("cvefiles", "fuzzerfiles") \
and not rel.endswith((".md", ".c")):
out.add(f"{corpus}/{rel}")
for f in sorted(out, key=lambda s: s.encode()):
print(f)
if __name__ == "__main__":
main(sys.argv[1])
conformance/probe/Cargo.lock
Generated
+492
View File
@@ -0,0 +1,492 @@
# This file is automatically @generated by Cargo.
# It is not intended for manual editing.
version = 4
[[package]]
name = "adler2"
version = "2.0.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "320119579fcad9c21884f5c4861d16174d0e06250625266f50fe6898340abefa"
[[package]]
name = "better_io"
version = "0.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ef0a3155e943e341e557863e69a708999c94ede624e37865c8e2a91b94efa78f"
[[package]]
name = "block-buffer"
version = "0.10.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3078c7629b62d3f0439517fa394996acacc5cbc91c5a20d8c658e77abd503a71"
dependencies = [
"generic-array",
]
[[package]]
name = "byteorder"
version = "1.5.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1fd0f2584146f6f2ef48085050886acf353beff7305ebd1ae69500e27c67f64b"
[[package]]
name = "bzip2"
version = "0.6.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f3a53fac24f34a81bc9954b5d6cfce0c21e18ec6959f44f56e8e90e4bb7c346c"
dependencies = [
"libbz2-rs-sys",
]
[[package]]
name = "cc"
version = "1.5.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f360145194ee8e21db5ee7f3fcd4fe52210864c75c985dae33218202c8bbe040"
dependencies = [
"find-msvc-tools",
"jobserver",
"libc",
"shlex",
]
[[package]]
name = "cfg-if"
version = "1.0.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4e7648175b45a9a48536d676f68d918270699102aa8dab5496df06904c914600"
[[package]]
name = "clawhdf5-format"
version = "2.7.0"
dependencies = [
"byteorder",
"bzip2",
"flate2",
"libaec-sys",
"libc",
"lz4_flex",
"pco",
"portable-atomic",
"ruzstd",
"sha2",
"snap",
"zstd",
]
[[package]]
name = "conformance-probe"
version = "0.1.0"
dependencies = [
"clawhdf5-format",
"serde_json",
"sha2",
]
[[package]]
name = "cpufeatures"
version = "0.2.17"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "59ed5838eebb26a2bb2e58f6d5b5316989ae9d08bab10e0e6d103e656d1b0280"
dependencies = [
"libc",
]
[[package]]
name = "crc32fast"
version = "1.5.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "01a7799fd6b852db0e61728dde9a204c423b44d689dbd432522543614b490e78"
dependencies = [
"cfg-if",
]
[[package]]
name = "crunchy"
version = "0.2.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "460fbee9c2c2f33933d720630a6a0bac33ba7053db5344fac858d4b8952d77d5"
[[package]]
name = "crypto-common"
version = "0.1.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "78c8292055d1c1df0cce5d180393dc8cce0abec0a7102adb6c7b1eef6016d60a"
dependencies = [
"generic-array",
"typenum",
]
[[package]]
name = "digest"
version = "0.10.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9ed9a281f7bc9b7576e61468ba615a66a5c8cfdff42420a70aa82701a3b1e292"
dependencies = [
"block-buffer",
"crypto-common",
]
[[package]]
name = "dtype_dispatch"
version = "0.2.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ab23e69df104e2fd85ee63a533a22d2132ef5975dc6b36f9f3e5a7305e4a8ed7"
[[package]]
name = "find-msvc-tools"
version = "0.1.14"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "aedcfb3409746eddb02b9e19ebda1c3394f759a152e48ee875a0844d1b955484"
[[package]]
name = "flate2"
version = "1.1.10"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6e634e2e0ebac1ee034020da1ca582e17ffe4e0f5e985823721e168928136dcb"
dependencies = [
"crc32fast",
"miniz_oxide",
"zlib-rs",
]
[[package]]
name = "generic-array"
version = "0.14.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "85649ca51fd72272d7821adaf274ad91c288277713d9c18820d8499a7ff69e9a"
dependencies = [
"typenum",
"version_check",
]
[[package]]
name = "getrandom"
version = "0.4.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "300e883d756b2e4ec94e02791f39b04b522276138852cfc41d9fb7e904106099"
dependencies = [
"cfg-if",
"libc",
"r-efi",
]
[[package]]
name = "half"
version = "2.7.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6ea2d84b969582b4b1864a92dc5d27cd2b77b622a8d79306834f1be5ba20d84b"
dependencies = [
"cfg-if",
"crunchy",
"zerocopy",
]
[[package]]
name = "itoa"
version = "1.0.18"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682"
[[package]]
name = "jobserver"
version = "0.1.35"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1c00acbd29eabad4a2392fa0e921c874934dbbf4194312ad20f04a0ed67a3cb3"
dependencies = [
"getrandom",
"libc",
]
[[package]]
name = "libaec-sys"
version = "0.1.0"
dependencies = [
"pkg-config",
]
[[package]]
name = "libbz2-rs-sys"
version = "0.2.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "34b357333733e8260735ba5894eb928c02ecc69c78715f01a8019e7fa7f2db4c"
[[package]]
name = "libc"
version = "0.2.189"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3eaf3ede3fee6db1a4c2ee091bf8a8b4dccdc6d17f656fb07896ee72867612f2"
[[package]]
name = "lz4_flex"
version = "0.11.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "373f5eceeeab7925e0c1098212f2fbc4d416adec9d35051a6ab251e824c1854a"
dependencies = [
"twox-hash",
]
[[package]]
name = "memchr"
version = "2.8.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "cf8baf1c55e62ffcace7a9f06f4bd9cd3f0c4beb022d3b367256b91b87513d98"
[[package]]
name = "miniz_oxide"
version = "0.9.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b63fbc4a50860e98e7b2aa7804ded1db5cbc3aff9193adaff57a6931bf7c4b4c"
dependencies = [
"adler2",
"simd-adler32",
]
[[package]]
name = "pco"
version = "1.0.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "386342cad4c6e97f081568e5d910ea7d871314c843aa8fc564f2a6b64cab9456"
dependencies = [
"better_io",
"dtype_dispatch",
"half",
"rand_xoshiro",
]
[[package]]
name = "pkg-config"
version = "0.3.34"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f6b464fbc74e149a392436b17d523f769e057cb6877f6a5c4618bc6f11800548"
[[package]]
name = "portable-atomic"
version = "1.15.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "05c8b63e8d9609db387f0324918f81d68fe27748f084ef092fb35954d0539a85"
[[package]]
name = "proc-macro2"
version = "1.0.107"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "985e7ec9bb745e6ce6535b544d84d6cd6f7ad8bd711c398938ae983b91a766d9"
dependencies = [
"unicode-ident",
]
[[package]]
name = "quote"
version = "1.0.47"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1fbf4db142a473a8d80c26bbf18454ed458bf8d26c8219c331daecfdbd079001"
dependencies = [
"proc-macro2",
]
[[package]]
name = "r-efi"
version = "6.0.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f8dcc9c7d52a811697d2151c701e0d08956f92b0e24136cf4cf27b57a6a0d9bf"
[[package]]
name = "rand_core"
version = "0.6.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ec0be4795e2f6a28069bec0b5ff3e2ac9bafc99e6a9a7dc3547996c5c816922c"
[[package]]
name = "rand_xoshiro"
version = "0.6.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6f97cdb2a36ed4183de61b2f824cc45c9f1037f28afe0a322e9fff4c108b5aaa"
dependencies = [
"rand_core",
]
[[package]]
name = "ruzstd"
version = "0.9.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a252f5e20f038fe7b4ea53e073e65398d652c864cc162fc77c56c2f13717b888"
dependencies = [
"twox-hash",
]
[[package]]
name = "serde"
version = "1.0.229"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4148590afebada386688f18773da617792bf2ef03ffc1e4cbd2b1d45b023e0ba"
dependencies = [
"serde_core",
]
[[package]]
name = "serde_core"
version = "1.0.229"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "67dca2c9c51e58a4791a4b1ed58308b39c64224d349a935ab5039aa360942a48"
dependencies = [
"serde_derive",
]
[[package]]
name = "serde_derive"
version = "1.0.229"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e7a5d71263a5a7d47b41f6b3f06ba276f10cc18b0931f1799f710578e2309348"
dependencies = [
"proc-macro2",
"quote",
"syn 3.0.6",
]
[[package]]
name = "serde_json"
version = "1.0.151"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c841b55ecdae098c80dcae9cf767f6f8a0c2cdb3416bbef72181df4d0fe73f14"
dependencies = [
"itoa",
"memchr",
"serde",
"serde_core",
"zmij",
]
[[package]]
name = "sha2"
version = "0.10.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a7507d819769d01a365ab707794a4084392c824f54a7a6a7862f8c3d0892b283"
dependencies = [
"cfg-if",
"cpufeatures",
"digest",
]
[[package]]
name = "shlex"
version = "2.0.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f8fadd59c855ef2080decdef8ff161eb6661b86933c9d82e5ba29dc602a55aba"
[[package]]
name = "simd-adler32"
version = "0.3.10"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3a219298ac11a56ea9a6d2120044824d6f01aeb034955e7af7bc16858527deea"
[[package]]
name = "snap"
version = "1.1.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "199905e6153d6405f9728fe44daace35f8f837bbf830bb6e85fbd5828709a886"
[[package]]
name = "syn"
version = "2.0.119"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "872831b642d1a07999a962a351ed35b955ea2cfc8f3862091e2a240a84f17297"
dependencies = [
"proc-macro2",
"quote",
"unicode-ident",
]
[[package]]
name = "syn"
version = "3.0.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8593e8e72159ed2257d083c7a454a85cbf854f37a0966d8d483aff8c8a3ebcee"
dependencies = [
"proc-macro2",
"quote",
"unicode-ident",
]
[[package]]
name = "twox-hash"
version = "2.1.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5283634e518fe9e82c7b20520bb4bc209009fd16c82077c802f8111ecbb0117a"
[[package]]
name = "typenum"
version = "1.20.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b6f5e870be6c3b371b77fe0ee0bafb859fa4964b4404c27de1d380043c4dda20"
[[package]]
name = "unicode-ident"
version = "1.0.26"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d245f478577f809a851594d02313b640fb437e0bb33866753cff937863096954"
[[package]]
name = "version_check"
version = "0.9.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0b928f33d975fc6ad9f86c8f283853ad26bdd5b10b7f1542aa2fa15e2289105a"
[[package]]
name = "zerocopy"
version = "0.8.59"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6df92bf3d9227be3d53173901ddbffac2babc27ae50f397776ffd6dc33f800cb"
dependencies = [
"zerocopy-derive",
]
[[package]]
name = "zerocopy-derive"
version = "0.8.59"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ac4f328cf2f05d084e496c3e9c3f33ed0a183656a16e1fcec4d464d8373aec82"
dependencies = [
"proc-macro2",
"quote",
"syn 2.0.119",
]
[[package]]
name = "zlib-rs"
version = "0.6.8"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b268e58e7c693d7c271f93ffc4ba3b380412554231c85bf61ca7af91042a4112"
[[package]]
name = "zmij"
version = "1.0.23"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "29666d0abbfad1e3dc4dcf6144730dd3a3ab225bbbdac83319345b1b44ccfc1b"
[[package]]
name = "zstd"
version = "0.13.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e91ee311a569c327171651566e07972200e76fcfe2242a4fa446149a3881c08a"
dependencies = [
"zstd-safe",
]
[[package]]
name = "zstd-safe"
version = "7.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "64d80649ab6db9d9f6f9c80a40becd948eda4714a0a5ac8c4d157a32231c7882"
dependencies = [
"zstd-sys",
]
[[package]]
name = "zstd-sys"
version = "2.1.0+zstd.1.5.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0ef0a8027ec3ee71300ab3bcbcd0393f434aa72b91ca6d635a39941deae8eea0"
dependencies = [
"cc",
"pkg-config",
]
conformance/probe/Cargo.toml
+25
View File
@@ -0,0 +1,25 @@
[package]
name = "conformance-probe"
version = "0.1.0"
edition = "2024"
rust-version = "1.92"
publish = false
description = "Walks an HDF5 file with clawhdf5-format and prints a canonical JSON description (see conformance/README.md)"
# Deliberately outside the main workspace: `cargo test --workspace` never
# builds it, and it links the optional C codecs (zstd, libaec) that the core
# crates' default build must not.
[workspace]
[dependencies]
clawhdf5-format = { path = "../../crates/clawhdf5-format", features = ["lz4", "zstd", "szip", "pcodec", "plugin-filters"] }
serde_json = "1"
sha2 = "0.10"
[profile.release]
# Keep panics catchable (the probe records them per object) and turn integer
# overflow into a reported panic instead of silent wraparound.
debug = 1
overflow-checks = true
debug-assertions = true
panic = "unwind"
conformance/probe/src/main.rs
+960
View File
@@ -0,0 +1,960 @@
//! Conformance probe: walks an HDF5 file with clawhdf5-format (the same calls
//! the `clawhdf5` facade makes) and prints a canonical JSON description:
//! every hard-linked object (sorted-name DFS, deduplicated by header address),
//! and for each dataset / attribute its shape plus the SHA-256 of its values
//! in a canonical encoding shared with `ref.py`.
//!
//! Canonical value encoding (per element, concatenated, row-major):
//! int / float / bitfield / enum / time : element bytes, little-endian
//! non-IEEE-layout float (e.g. N-Bit) : the IEEE float of the same size it converts to
//! int with bit offset / short precision: the full-width integer it converts to
//! opaque : raw bytes
//! compound : members in declaration order (padding dropped)
//! array : base elements row-major
//! string (fixed or VL) : b'S' + u32le len + bytes (cut at first NUL, trailing spaces stripped)
//! VL sequence : b'V' + u32le count + base elements
//! reference : b'R' (payload not compared)
//!
//! Every object is processed inside catch_unwind; a caught panic is recorded
//! with its message, location and the clawhdf5 frames of its backtrace.
use std::cell::RefCell;
use std::collections::HashSet;
use std::panic::{self, AssertUnwindSafe};
use clawhdf5_format::attribute::extract_attributes_full;
use clawhdf5_format::data_layout::DataLayout;
use clawhdf5_format::data_read;
use clawhdf5_format::dataspace::{Dataspace, DataspaceType};
use clawhdf5_format::datatype::{Datatype, DatatypeByteOrder};
use clawhdf5_format::filter_pipeline::FilterPipeline;
use clawhdf5_format::group_v1::{self, GroupEntry};
use clawhdf5_format::group_v2;
use clawhdf5_format::message_type::MessageType;
use clawhdf5_format::object_header::{ObjectClass, ObjectHeader};
use clawhdf5_format::signature;
use clawhdf5_format::superblock::Superblock;
use clawhdf5_format::symbol_table::SymbolTableMessage;
use clawhdf5_format::vl_data::{VlResolver, check_element_size};
use serde_json::{Map, Value, json};
use sha2::{Digest, Sha256};
const MAX_BYTES: u64 = 200 * 1024 * 1024;
const MAX_OBJECTS: usize = 200_000;
thread_local! {
static LAST_PANIC: RefCell<Option<String>> = const { RefCell::new(None) };
}
fn install_hook() {
panic::set_hook(Box::new(|info| {
let msg = if let Some(s) = info.payload().downcast_ref::<&str>() {
s.to_string()
} else if let Some(s) = info.payload().downcast_ref::<String>() {
s.clone()
} else {
"<non-string panic>".into()
};
let loc = info
.location()
.map(|l| format!("{}:{}", l.file(), l.line()))
.unwrap_or_default();
let bt = std::backtrace::Backtrace::force_capture().to_string();
// keep only frames from clawhdf5 code
let mut frames = Vec::new();
let lines: Vec<&str> = bt.lines().collect();
for (i, l) in lines.iter().enumerate() {
let t = l.trim();
if t.contains("clawhdf5_format::") || t.contains("conformance_probe::") {
let at = lines
.get(i + 1)
.map(|n| n.trim())
.filter(|n| n.starts_with("at "))
.map(|n| {
let n = n.trim_start_matches("at ");
match n.find("/crates/") {
Some(p) => n[p + 1..].to_string(),
None => n.to_string(),
}
})
.unwrap_or_default();
let name = t.split_once(": ").map(|x| x.1).unwrap_or(t);
frames.push(format!("{name} ({at})"));
if frames.len() >= 12 {
break;
}
}
}
let full = format!("PANIC: {msg} @ {loc}\n {}", frames.join("\n "));
eprintln!("{full}");
LAST_PANIC.with(|p| *p.borrow_mut() = Some(full));
}));
}
/// Run `f`, turning a panic into Err("PANIC: ...").
fn guarded<T>(f: impl FnOnce() -> Result<T, String>) -> Result<T, String> {
match panic::catch_unwind(AssertUnwindSafe(f)) {
Ok(r) => r,
Err(_) => Err(LAST_PANIC
.with(|p| p.borrow_mut().take())
.unwrap_or_else(|| "PANIC: <unknown>".into())),
}
}
fn e<E: std::fmt::Debug>(x: E) -> String {
format!("{x:?}")
}
struct Ctx<'a> {
data: &'a [u8],
os: u8,
ls: u8,
base_dir: std::path::PathBuf,
/// Resolves variable-length elements as the library does (null
/// elements, strings cut at a NUL, heap objects of the wrong size
/// refused), caching each heap collection.
vl: RefCell<VlResolver<'a>>,
}
impl<'a> Ctx<'a> {
fn header(&self, addr: u64) -> Result<ObjectHeader, String> {
ObjectHeader::parse(self.data, addr as usize, self.os, self.ls).map_err(e)
}
fn payload(&self, h: &ObjectHeader, t: MessageType) -> Result<Option<Vec<u8>>, String> {
match h.messages.iter().find(|m| m.msg_type == t) {
None => Ok(None),
Some(m) => {
clawhdf5_format::shared_message::message_data(self.data, m, self.os, self.ls)
.map(|c| Some(c.into_owned()))
.map_err(e)
}
}
}
fn canon(&self, dt: &Datatype, b: &[u8], out: &mut Vec<u8>) -> Result<(), String> {
let size = dt.type_size() as usize;
if b.len() < size {
return Err(format!(
"canon: element slice {} < type size {size}",
b.len()
));
}
match dt {
Datatype::FloatingPoint { .. } if !ieee_layout(dt) => {
canon_custom_float(dt, &b[..size], out)?
}
Datatype::FixedPoint { .. } if partial_int(dt) => {
canon_partial_int(dt, &b[..size], out)?
}
Datatype::FixedPoint { byte_order, .. }
| Datatype::BitField { byte_order, .. }
| Datatype::FloatingPoint { byte_order, .. } => match byte_order {
DatatypeByteOrder::LittleEndian => out.extend_from_slice(&b[..size]),
DatatypeByteOrder::BigEndian => out.extend(b[..size].iter().rev()),
DatatypeByteOrder::Vax => return Err("canon: VAX byte order".into()),
},
Datatype::Time { .. } | Datatype::Opaque { .. } => out.extend_from_slice(&b[..size]),
Datatype::String { .. } => canon_str(&b[..size], out),
Datatype::Compound { members, .. } => {
for m in members {
let off = m.byte_offset as usize;
let ms = m.datatype.type_size() as usize;
if off.checked_add(ms).is_none_or(|end| end > size) {
return Err(format!("canon: member {} out of bounds", m.name));
}
self.canon(&m.datatype, &b[off..off + ms], out)?;
}
}
Datatype::Reference { .. } => out.push(b'R'),
Datatype::Enumeration { base_type, .. } => self.canon(base_type, b, out)?,
Datatype::Array {
base_type,
dimensions,
} => {
let n: usize = dimensions.iter().map(|d| *d as usize).product();
let bs = base_type.type_size() as usize;
for i in 0..n {
self.canon(base_type, &b[i * bs..], out)?;
}
}
Datatype::VariableLength {
size: vl_size,
is_string,
base_type,
..
} => {
check_element_size(*vl_size, self.os).map_err(e)?;
let el = &b[..size];
if *is_string {
let s = self.vl.borrow_mut().string_bytes(el).map_err(e)?;
canon_str(&s[0], out);
} else {
let bs = base_type.type_size() as usize;
// The borrow ends here: the base type may itself be
// variable-length.
let seq = self.vl.borrow_mut().sequences(el, bs).map_err(e)?;
let seq = &seq[0];
let len = seq.len() / bs;
out.push(b'V');
out.extend_from_slice(&(len as u32).to_le_bytes());
for i in 0..len {
self.canon(base_type, &seq[i * bs..], out)?;
}
}
}
}
Ok(())
}
/// Returns (shape json, n_elements)
fn shape(ds: &Dataspace) -> (Value, u64) {
match ds.space_type {
DataspaceType::Null => (Value::String("null".into()), 0),
DataspaceType::Scalar => (json!([]), 1),
DataspaceType::Simple => {
let n = ds.dimensions.iter().fold(1u64, |a, d| a.saturating_mul(*d));
(json!(ds.dimensions), n)
}
}
}
fn hash_values(
&self,
dt: &Datatype,
raw: &[u8],
n: u64,
rec: &mut Map<String, Value>,
) -> Result<(), String> {
let size = dt.type_size() as usize;
let need = (n as usize).checked_mul(size).ok_or("n*size overflow")?;
if raw.len() != need {
return Err(format!(
"raw length {} != n_elements {n} * type_size {size}",
raw.len()
));
}
let mut canon = Vec::with_capacity(need);
for i in 0..n as usize {
self.canon(dt, &raw[i * size..(i + 1) * size], &mut canon)?;
}
let h = Sha256::digest(&canon);
rec.insert("hash".into(), Value::String(hex(&h)));
rec.insert(
"head".into(),
Value::String(hex(&canon[..canon.len().min(48)])),
);
Ok(())
}
/// VDS source files resolve next to the virtual file; like the library,
/// refuse absolute paths and `..`.
fn vds_resolver(
&self,
) -> impl Fn(&str) -> Result<Option<Vec<u8>>, clawhdf5_format::error::FormatError> + use<> {
let base = self.base_dir.clone();
move |name: &str| {
use clawhdf5_format::error::FormatError;
let p = std::path::Path::new(name);
if p.is_absolute()
|| p.components()
.any(|c| matches!(c, std::path::Component::ParentDir))
{
return Err(FormatError::ChunkedReadError(format!("refused {name}")));
}
match std::fs::read(base.join(p)) {
Ok(b) => Ok(Some(b)),
Err(err) if err.kind() == std::io::ErrorKind::NotFound => Ok(None),
Err(err) => Err(FormatError::ChunkedReadError(err.to_string())),
}
}
}
fn read_named_datatype(&self, h: &ObjectHeader) -> Result<(), String> {
let dtb = self
.payload(h, MessageType::Datatype)?
.ok_or("MissingMessage(Datatype)")?;
Datatype::parse_in_header(&dtb, h.version).map_err(e)?;
Ok(())
}
fn read_dataset(&self, h: &ObjectHeader, rec: &mut Map<String, Value>) -> Result<(), String> {
let dtb = self
.payload(h, MessageType::Datatype)?
.ok_or("MissingMessage(Datatype)")?;
let (dt, _) = Datatype::parse_in_header(&dtb, h.version).map_err(e)?;
rec.insert("dtype".into(), Value::String(dtype_str(&dt)));
let dsb = self
.payload(h, MessageType::Dataspace)?
.ok_or("MissingMessage(Dataspace)")?;
let mut ds = Dataspace::parse(&dsb, self.ls).map_err(e)?;
// A virtual dataset's extent can come from its sources (unlimited /
// printf mappings), as h5py reports it, rather than the stored one.
if let Some(lm) = h
.messages
.iter()
.find(|m| m.msg_type == MessageType::DataLayout)
&& let Ok(dl @ DataLayout::Virtual { .. }) =
DataLayout::parse(&lm.data, self.os, self.ls)
{
let resolver = self.vds_resolver();
ds.dimensions = clawhdf5_format::vds::virtual_dataset_extent(
self.data,
&dl,
&ds,
self.os,
self.ls,
Some(&resolver),
)
.map_err(e)?;
}
let lm = h
.messages
.iter()
.find(|m| m.msg_type == MessageType::DataLayout)
.ok_or("MissingMessage(DataLayout)")?;
let dl = DataLayout::parse(&lm.data, self.os, self.ls).map_err(e)?;
// What libhdf5 checks when it opens the dataset (as File::dataset).
data_read::check_dataset_storage(&dl, &ds, &dt, self.data.len() as u64).map_err(e)?;
let (shape, n) = Self::shape(&ds);
rec.insert("shape".into(), shape);
if n.saturating_mul(dt.type_size() as u64) > MAX_BYTES {
rec.insert("skipped".into(), Value::String("too large".into()));
return Ok(());
}
rec.insert(
"layout".into(),
Value::String(
match &dl {
DataLayout::Compact { .. } => "compact",
DataLayout::Contiguous { .. } => "contiguous",
DataLayout::Chunked { .. } => "chunked",
DataLayout::Virtual { .. } => "virtual",
}
.into(),
),
);
let pipeline = match self.payload(h, MessageType::FilterPipeline)? {
Some(p) => Some(FilterPipeline::parse(&p).map_err(e)?),
None => None,
};
if let Some(p) = &pipeline {
rec.insert(
"filters".into(),
json!(p.filters.iter().map(|f| f.filter_id).collect::<Vec<_>>()),
);
}
let raw = if matches!(dl, DataLayout::Virtual { .. }) {
let resolver = self.vds_resolver();
let fill = clawhdf5_format::fill_value::dataset_fill_value_in(
self.data,
&h.messages,
self.os,
self.ls,
)
.map_err(e)?;
clawhdf5_format::vds::read_virtual_dataset(
self.data,
&dl,
&ds,
&dt,
fill.as_deref(),
self.os,
self.ls,
Some(&resolver),
)
.map_err(e)?
.data
} else {
let cache = clawhdf5_format::chunk_cache::ChunkCache::new();
clawhdf5_format::fill_value::read_full_with_fill::<clawhdf5_format::error::FormatError>(
&h.messages,
self.data,
&dl,
&ds,
dt.type_size() as usize,
self.os,
self.ls,
|| {
data_read::read_raw_data_cached(
self.data,
&dl,
&ds,
&dt,
pipeline.as_ref(),
self.os,
self.ls,
&cache,
)
},
)
.map_err(e)?
};
self.hash_values(&dt, &raw, n, rec)
}
fn attrs(&self, h: &ObjectHeader) -> Result<Map<String, Value>, String> {
let msgs = extract_attributes_full(self.data, h, self.os, self.ls).map_err(e)?;
let mut out = Map::new();
for a in &msgs {
let r = guarded(|| {
let mut rec = Map::new();
rec.insert("dtype".into(), Value::String(dtype_str(&a.datatype)));
let (shape, n) = Self::shape(&a.dataspace);
rec.insert("shape".into(), shape);
self.hash_values(&a.datatype, &a.raw_data, n, &mut rec)?;
Ok(rec)
});
let v = match r {
Ok(rec) => Value::Object(rec),
Err(msg) => json!({ "error": msg }),
};
out.insert(a.name.clone(), v);
}
Ok(out)
}
fn entries(&self, h: &ObjectHeader) -> Result<Vec<GroupEntry>, String> {
let v1 = h
.messages
.iter()
.find(|m| m.msg_type == MessageType::SymbolTable);
if let Some(m) = v1 {
let stm = SymbolTableMessage::parse(&m.data, self.os).map_err(e)?;
group_v1::resolve_v1_group_entries(self.data, &stm, self.os, self.ls).map_err(e)
} else if h
.messages
.iter()
.any(|m| m.msg_type == MessageType::LinkInfo || m.msg_type == MessageType::Link)
{
group_v2::resolve_v2_group_entries(self.data, h, self.os, self.ls).map_err(e)
} else {
Ok(Vec::new())
}
}
}
/// Element bytes as an unsigned integer (at most 16 bytes), honouring byte order.
fn element_bits(b: &[u8], byte_order: &DatatypeByteOrder) -> Result<u128, String> {
if b.len() > 16 {
return Err(format!("canon: {}-byte numeric element", b.len()));
}
let mut v = 0u128;
match byte_order {
DatatypeByteOrder::LittleEndian => {
for (i, x) in b.iter().enumerate() {
v |= u128::from(*x) << (8 * i);
}
}
DatatypeByteOrder::BigEndian => {
for x in b {
v = (v << 8) | u128::from(*x);
}
}
DatatypeByteOrder::Vax => return Err("canon: VAX byte order".into()),
}
Ok(v)
}
fn field(v: u128, pos: u32, len: u32) -> u128 {
if len == 0 || pos >= 128 {
return 0;
}
let v = v >> pos;
if len >= 128 {
v
} else {
v & ((1u128 << len) - 1)
}
}
/// True when a float's bit fields are exactly IEEE 754 binary16/32/64 for its
/// size. h5py hands back such a type's bytes untouched; any other layout (an
/// N-Bit `H5Tset_precision` float, say) is *converted* by libhdf5 into the
/// numpy float of the same size, so comparing raw bytes would be meaningless.
fn ieee_layout(dt: &Datatype) -> bool {
let Datatype::FloatingPoint {
size,
bit_offset,
bit_precision,
exponent_location,
exponent_size,
mantissa_location,
mantissa_size,
exponent_bias,
..
} = dt
else {
return true;
};
let std = match size {
2 => (16, 10, 5, 10, 15),
4 => (32, 23, 8, 23, 127),
8 => (64, 52, 11, 52, 1023),
_ => return true, // no same-size numpy float to convert to: compare raw
};
*bit_offset == 0
&& (
*bit_precision,
*exponent_location,
*exponent_size,
*mantissa_size,
*exponent_bias,
) == (std.0, std.1, std.2, std.3, std.4)
&& *mantissa_location == 0
}
/// Canonicalise a non-IEEE-layout float the way libhdf5's float->float
/// conversion presents it to h5py: as the IEEE float of the same size.
/// Assumes the implied-leading-one normalisation and the sign bit at the top
/// of the precision (what `H5Tset_precision` produces; the parser does not
/// keep either field).
fn canon_custom_float(dt: &Datatype, b: &[u8], out: &mut Vec<u8>) -> Result<(), String> {
let Datatype::FloatingPoint {
size,
byte_order,
bit_offset,
bit_precision,
exponent_location,
exponent_size,
mantissa_location,
mantissa_size,
exponent_bias,
} = dt
else {
unreachable!()
};
let (esize, msize) = (u32::from(*exponent_size), u32::from(*mantissa_size));
if esize == 0 || esize > 30 || msize > 64 {
return Err(format!("canon: unsupported float layout e{esize} m{msize}"));
}
let v = element_bits(b, byte_order)?;
let sign_pos = (u32::from(*bit_offset) + u32::from(*bit_precision)).saturating_sub(1);
let neg = field(v, sign_pos, 1) == 1;
let e = field(v, u32::from(*exponent_location), esize) as i64;
let m = field(v, u32::from(*mantissa_location), msize);
let emax = (1i64 << esize) - 1;
let bias = i64::from(*exponent_bias);
let mag = if e == emax {
if m == 0 { f64::INFINITY } else { f64::NAN }
} else if e == 0 {
(m as f64) * 2f64.powi((1 - bias - msize as i64) as i32)
} else {
((1u128 << msize) as f64 + m as f64) * 2f64.powi((e - bias - msize as i64) as i32)
};
let x = if neg { -mag } else { mag };
match size {
2 => out
.extend_from_slice(&clawhdf5_format::float16::f32_to_f16_bits(x as f32).to_le_bytes()),
4 => out.extend_from_slice(&(x as f32).to_le_bytes()),
8 => out.extend_from_slice(&x.to_le_bytes()),
_ => unreachable!("ieee_layout keeps other sizes raw"),
}
Ok(())
}
/// Integers stored with a bit offset or reduced precision (N-Bit): libhdf5
/// converts them to the full-width integer of the same size, shifting the
/// value down and sign-extending from the top precision bit.
fn canon_partial_int(dt: &Datatype, b: &[u8], out: &mut Vec<u8>) -> Result<(), String> {
let Datatype::FixedPoint {
size,
byte_order,
signed,
bit_offset,
bit_precision,
} = dt
else {
unreachable!()
};
let prec = u32::from(*bit_precision);
let v = element_bits(b, byte_order)?;
let mut x = field(v, u32::from(*bit_offset), prec);
if *signed && prec > 0 && prec < 128 && field(x, prec - 1, 1) == 1 {
x |= !0u128 << prec;
}
out.extend_from_slice(&x.to_le_bytes()[..*size as usize]);
Ok(())
}
fn partial_int(dt: &Datatype) -> bool {
matches!(dt, Datatype::FixedPoint { size, bit_offset, bit_precision, .. }
if *bit_offset != 0 || u32::from(*bit_precision) != size * 8)
}
fn canon_str(b: &[u8], out: &mut Vec<u8>) {
let cut = b.iter().position(|&c| c == 0).unwrap_or(b.len());
let mut s = &b[..cut];
while let [rest @ .., b' '] = s {
s = rest;
}
out.push(b'S');
out.extend_from_slice(&(s.len() as u32).to_le_bytes());
out.extend_from_slice(s);
}
fn hex(b: &[u8]) -> String {
b.iter().map(|x| format!("{x:02x}")).collect()
}
fn dtype_str(dt: &Datatype) -> String {
match dt {
Datatype::FixedPoint {
size,
signed,
byte_order,
..
} => {
format!(
"{}{}{}",
bo(byte_order),
if *signed { "i" } else { "u" },
size
)
}
Datatype::FloatingPoint {
size, byte_order, ..
} => format!("{}f{}", bo(byte_order), size),
Datatype::BitField {
size, byte_order, ..
} => format!("{}b{}", bo(byte_order), size),
Datatype::Time { size, .. } => format!("time{size}"),
Datatype::String { size, .. } => format!("S{size}"),
Datatype::Opaque { size, .. } => format!("V{size}"),
Datatype::Compound { size, members } => format!(
"{{{}}}{size}",
members
.iter()
.map(|m| format!("{}:{}", m.name, dtype_str(&m.datatype)))
.collect::<Vec<_>>()
.join(",")
),
Datatype::Reference { ref_type, .. } => format!("ref({ref_type:?})"),
Datatype::Enumeration { base_type, .. } => format!("enum({})", dtype_str(base_type)),
Datatype::VariableLength {
is_string: true, ..
} => "vlstr".into(),
Datatype::VariableLength { base_type, .. } => format!("vlen({})", dtype_str(base_type)),
Datatype::Array {
base_type,
dimensions,
} => format!("({}){dimensions:?}", dtype_str(base_type)),
}
}
fn bo(b: &DatatypeByteOrder) -> &'static str {
match b {
DatatypeByteOrder::LittleEndian => "<",
DatatypeByteOrder::BigEndian => ">",
DatatypeByteOrder::Vax => "vax",
}
}
fn is_group(h: &ObjectHeader) -> bool {
h.messages.iter().any(|m| {
matches!(
m.msg_type,
MessageType::LinkInfo | MessageType::Link | MessageType::SymbolTable
)
})
}
/// The probe's kind for an object header: libhdf5's object class
/// ([`ObjectHeader::object_class`]: group, then dataset — a datatype *and* a
/// dataspace — then named datatype), which is what h5py opens the object as.
/// The root group, and a header with only link messages, count as groups.
fn kind_of(h: &ObjectHeader, is_root: bool) -> &'static str {
match h.object_class() {
Some(ObjectClass::Group) => "group",
Some(ObjectClass::Dataset) => "dataset",
_ if is_root || is_group(h) => "group",
Some(ObjectClass::NamedDatatype) => "datatype",
None => "unknown",
}
}
fn main() {
install_hook();
let path = std::env::args().nth(1).expect("usage: probe <file>");
let mut top = Map::new();
top.insert("file".into(), Value::String(path.clone()));
let data = match std::fs::read(&path) {
Ok(d) => d,
Err(err) => {
top.insert("open_error".into(), Value::String(format!("Io({err})")));
println!("{}", Value::Object(top));
return;
}
};
// Every address is relative to the superblock: look at the file from
// there on (past any user block), as libhdf5 does.
let hdf5: &[u8] = match signature::find_signature(&data) {
Ok(off) => &data[off..],
Err(_) => &data,
};
let sb = guarded(|| Superblock::parse(hdf5, 0).map_err(e));
let sb = match sb {
Ok(sb) => sb,
Err(msg) => {
top.insert("open_error".into(), Value::String(msg));
println!("{}", Value::Object(top));
return;
}
};
// libhdf5 refuses a truncated file and reads nothing past the recorded
// end of file.
let base = (data.len() - hdf5.len()) as u64;
let hdf5 = match sb.data_end(base, data.len() as u64) {
Ok(end) => &hdf5[..end as usize],
Err(err) => {
top.insert("open_error".into(), Value::String(e(err)));
println!("{}", Value::Object(top));
return;
}
};
// libhdf5 decodes the superblock extension at open (an error refuses
// the file), and loads a metadata cache image over the file's own
// metadata. It loads the image only when it first reads metadata — the
// root group — so a file whose image it cannot load still opens and
// that read fails. The library decides all three cases with the same
// `cache_image_state`: `File` and `MmapFile` open such a file and fail
// every object lookup with the image's error, which is what the probe
// records here (on the root group, where libhdf5 reports it).
use clawhdf5_format::superblock_ext::{self, CacheImageState};
let state = match guarded(|| superblock_ext::cache_image_state(hdf5, &sb).map_err(e)) {
Ok(x) => x,
Err(msg) => {
top.insert("open_error".into(), Value::String(msg));
println!("{}", Value::Object(top));
return;
}
};
let mut image_error = None;
let view = match state {
CacheImageState::Absent => None,
CacheImageState::Unloadable(err) => {
image_error = Some(e(err));
None
}
CacheImageState::Loaded(image) => {
let mut v = hdf5.to_vec();
match image.block(hdf5).and_then(|b| image.apply(b, &mut v)) {
Ok(()) => Some(v),
Err(err) => {
image_error = Some(e(err));
None
}
}
}
};
let hdf5: &[u8] = view.as_deref().unwrap_or(hdf5);
top.insert("superblock_version".into(), json!(sb.version));
let ctx = Ctx {
data: hdf5,
os: sb.offset_size,
ls: sb.length_size,
base_dir: std::path::Path::new(&path)
.parent()
.map(|p| p.to_path_buf())
.unwrap_or_default(),
vl: RefCell::new(VlResolver::new(hdf5, sb.offset_size, sb.length_size)),
};
let mut objects: Vec<Value> = Vec::new();
let mut visited = HashSet::new();
let mut soft_v1 = 0u64;
// explicit DFS stack: (address, path)
let mut stack: Vec<(u64, String)> = vec![(sb.root_group_address, "/".to_string())];
while let Some((addr, p)) = stack.pop() {
if objects.len() >= MAX_OBJECTS {
top.insert("truncated".into(), json!(true));
break;
}
if !visited.insert(addr) {
continue;
}
let mut rec = Map::new();
rec.insert("path".into(), Value::String(p.clone()));
let r = guarded(|| {
if let Some(msg) = &image_error {
return Err(msg.clone());
}
let h = ctx.header(addr)?;
Ok(h)
});
let h = match r {
Ok(h) => h,
Err(msg) => {
rec.insert("kind".into(), Value::String("unknown".into()));
rec.insert("error".into(), Value::String(msg));
objects.push(Value::Object(rec));
continue;
}
};
let kind = kind_of(&h, addr == sb.root_group_address);
rec.insert("kind".into(), Value::String(kind.into()));
if kind == "dataset"
&& let Err(msg) = guarded(|| ctx.read_dataset(&h, &mut rec))
{
rec.insert("error".into(), Value::String(msg));
}
// Opening a committed datatype decodes it (h5py's `f[name]` fails on
// one libhdf5 cannot decode), so decode it here too.
if kind == "datatype"
&& let Err(msg) = guarded(|| ctx.read_named_datatype(&h))
{
rec.insert("error".into(), Value::String(msg));
}
if kind != "datatype" {
match guarded(|| ctx.attrs(&h)) {
Ok(m) => {
rec.insert("attrs".into(), Value::Object(m));
}
Err(msg) => {
rec.insert("attrs_error".into(), Value::String(msg));
}
}
}
if kind == "group" {
match guarded(|| ctx.entries(&h)) {
Ok(mut ents) => {
ents.retain(|en| {
if en.cache_type == 2 {
soft_v1 += 1;
false
} else {
true
}
});
ents.sort_by(|a, b| a.name.cmp(&b.name));
let base = if p == "/" { String::new() } else { p.clone() };
for en in ents.into_iter().rev() {
stack.push((en.object_header_address, format!("{base}/{}", en.name)));
}
}
Err(msg) => {
rec.insert("list_error".into(), Value::String(msg));
}
}
}
objects.push(Value::Object(rec));
}
if soft_v1 > 0 {
top.insert("v1_soft_link_entries".into(), json!(soft_v1));
}
top.insert("objects".into(), Value::Array(objects));
println!("{}", Value::Object(top));
}
#[cfg(test)]
mod tests {
use super::*;
/// The N-Bit float of libhdf5's `test/testfiles/le_data.h5`
/// (`Nbit_float_data_le`): offset 7, precision 20, sign bit 26, exponent
/// 20+6 (bias 31), mantissa 7+13.
fn nbit_f32(byte_order: DatatypeByteOrder) -> Datatype {
Datatype::FloatingPoint {
size: 4,
byte_order,
bit_offset: 7,
bit_precision: 20,
exponent_location: 20,
exponent_size: 6,
mantissa_location: 7,
mantissa_size: 13,
exponent_bias: 31,
}
}
fn canon_one(dt: &Datatype, bytes: &[u8]) -> Vec<u8> {
let mut out = Vec::new();
canon_custom_float(dt, bytes, &mut out).unwrap();
out
}
#[test]
fn nbit_float_canonicalises_to_the_value_libhdf5_returns() {
let le = nbit_f32(DatatypeByteOrder::LittleEndian);
let be = nbit_f32(DatatypeByteOrder::BigEndian);
assert!(!ieee_layout(&le));
// 1.0: exponent = bias, mantissa 0
let one: u32 = 31 << 20;
assert_eq!(canon_one(&le, &one.to_le_bytes()), 1.0f32.to_le_bytes());
assert_eq!(canon_one(&be, &one.to_be_bytes()), 1.0f32.to_le_bytes());
// -2.1999512 (h5py's reading of the file's -2.2): sign, e = 32, m = 819
let v: u32 = (1 << 26) | (32 << 20) | (819 << 7);
assert_eq!(
canon_one(&le, &v.to_le_bytes()),
(-2.199_951_2f32).to_le_bytes()
);
assert_eq!(canon_one(&le, &[0; 4]), 0.0f32.to_le_bytes());
}
#[test]
fn ieee_floats_keep_their_raw_bytes() {
let f32le = Datatype::FloatingPoint {
size: 4,
byte_order: DatatypeByteOrder::LittleEndian,
bit_offset: 0,
bit_precision: 32,
exponent_location: 23,
exponent_size: 8,
mantissa_location: 0,
mantissa_size: 23,
exponent_bias: 127,
};
assert!(ieee_layout(&f32le));
}
#[test]
fn kind_follows_libhdf5_object_class() {
use clawhdf5_format::object_header::HeaderMessage;
let header = |types: &[MessageType]| ObjectHeader {
version: 2,
messages: types
.iter()
.map(|&msg_type| HeaderMessage {
msg_type,
size: 0,
flags: 0,
creation_order: None,
data: Vec::new(),
})
.collect(),
reference_count: None,
flags: 0,
access_time: None,
modification_time: None,
change_time: None,
birth_time: None,
};
use MessageType::*;
// cve-2024-33874 `/Dset1`: a datatype and a layout but no dataspace
// is a named datatype to libhdf5 (h5py opens it as one).
assert_eq!(kind_of(&header(&[Datatype, DataLayout]), false), "datatype");
assert_eq!(
kind_of(&header(&[Datatype, Dataspace, DataLayout]), false),
"dataset"
);
assert_eq!(kind_of(&header(&[SymbolTable]), false), "group");
assert_eq!(kind_of(&header(&[Link]), false), "group");
assert_eq!(kind_of(&header(&[]), true), "group");
assert_eq!(kind_of(&header(&[]), false), "unknown");
}
#[test]
fn partial_precision_int_is_shifted_and_sign_extended() {
let dt = Datatype::FixedPoint {
size: 4,
byte_order: DatatypeByteOrder::BigEndian,
signed: true,
bit_offset: 4,
bit_precision: 17,
};
assert!(partial_int(&dt));
let stored = (((-5i32) as u32) & 0x1_FFFF) << 4;
let mut out = Vec::new();
canon_partial_int(&dt, &stored.to_be_bytes(), &mut out).unwrap();
assert_eq!(out, (-5i32).to_le_bytes());
}
}
conformance/ref.py
Executable
+273
View File
@@ -0,0 +1,273 @@
#!/usr/bin/env python3
"""Reference probe: same JSON as the Rust `conformance-probe`, produced with h5py.
Walk: iterative DFS from '/', children in sorted (UTF-8 byte) name order, hard
links only, each object once (first path wins, deduplicated by object identity).
Canonical value encoding: see harness/src/main.rs.
"""
import hashlib
import json
import os
import struct
import sys
import numpy as np
import h5py
try:
import hdf5plugin # noqa: F401 registers blosc/lz4/zstd/bzip2/... filters
except Exception: # pragma: no cover
pass
MAX_BYTES = 200 * 1024 * 1024
MAX_OBJECTS = 200_000
def canon_str(b, out):
if isinstance(b, str):
b = b.encode("utf-8", "surrogateescape")
b = bytes(b)
cut = b.find(b"\x00")
if cut >= 0:
b = b[:cut]
b = b.rstrip(b" ")
out += b"S" + struct.pack("<I", len(b)) + b
def simple(dt):
if dt.fields:
return all(simple(dt.fields[n][0]) for n in dt.names)
if dt.subdtype:
return simple(dt.subdtype[0])
return dt.kind in "iufcbV"
def packed(dt):
if dt.fields:
return np.dtype([(n, packed(dt.fields[n][0])) for n in dt.names])
if dt.subdtype:
base, shape = dt.subdtype
return np.dtype((packed(base), shape))
if dt.kind in "iufcb":
return dt.newbyteorder("<")
return dt
def canon_el(dt, val, out):
if dt.fields:
for n in dt.names:
canon_el(dt.fields[n][0], val[n], out)
return
if dt.subdtype:
base, _ = dt.subdtype
for x in np.asarray(val).reshape(-1):
canon_el(base, x, out)
return
k = dt.kind
if k in "iufcb":
out += np.asarray(val, dtype=dt).astype(dt.newbyteorder("<")).tobytes()
elif k == "V":
out += np.asarray(val, dtype=dt).tobytes()
elif k == "S":
canon_str(val, out)
elif k == "O":
if h5py.check_string_dtype(dt) is not None:
canon_str(val if val is not None else b"", out)
elif h5py.check_ref_dtype(dt) is not None:
out += b"R"
else:
base = h5py.check_vlen_dtype(dt)
if base is None:
raise TypeError(f"unhandled object dtype {dt!r}")
arr = np.asarray(val if val is not None else [], dtype=base).reshape(-1)
out += b"V" + struct.pack("<I", arr.shape[0])
if simple(base):
out += arr.astype(packed(base)).tobytes()
else:
for x in arr:
canon_el(base, x, out)
elif k == "U":
canon_str(str(val), out)
else:
raise TypeError(f"unhandled dtype kind {k} ({dt!r})")
def has_obj(dt):
if dt.fields:
return any(has_obj(dt.fields[n][0]) for n in dt.names)
if dt.subdtype:
return has_obj(dt.subdtype[0])
return dt.kind == "O"
def note_conversion(tid, dt, rec):
"""h5py converts some file types (FP8, bfloat16, x87 long double, ...) to a
different-sized numpy type; then value bytes are not comparable."""
try:
if not has_obj(dt) and tid.get_size() != dt.itemsize:
rec["converted"] = f"file type size {tid.get_size()} -> numpy {dt} ({dt.itemsize})"
except Exception: # noqa: BLE001
pass
def hash_values(arr, dt, rec):
# h5py expands an HDF5 array element type into trailing array dims, a
# nested array type (an array of arrays) into all of them. Converting the
# expanded array back to the inner subarray type would broadcast every
# element into a whole subarray, so strip every level.
while dt.subdtype is not None:
dt = dt.subdtype[0]
arr = np.asarray(arr, dtype=dt)
if simple(dt):
c = np.ascontiguousarray(arr).astype(packed(dt)).tobytes()
else:
out = bytearray()
for x in arr.reshape(-1):
canon_el(dt, x, out)
c = bytes(out)
rec["hash"] = hashlib.sha256(c).hexdigest()
rec["head"] = c[:48].hex()
def err(e):
s = f"{type(e).__name__}: {e}"
return s.splitlines()[0][:400] if s else type(e).__name__
def shape_of(s):
return "null" if s is None else list(s)
def n_bytes(shape, tid):
n = 1
for d in shape or ():
n *= d
return n * tid.get_size()
def read_attrs(obj):
out = {}
names = sorted(obj.attrs.keys(), key=lambda s: s.encode("utf-8", "surrogateescape"))
for name in names:
rec = {}
try:
aid = obj.attrs.get_id(name)
rec["dtype"] = str(aid.dtype)
rec["shape"] = shape_of(aid.shape)
note_conversion(aid.get_type(), aid.dtype, rec)
if aid.shape is None:
hash_values(np.empty((0,), dtype=aid.dtype), aid.dtype, rec)
else:
val = obj.attrs[name]
hash_values(val, aid.dtype, rec)
except Exception as e: # noqa: BLE001
rec = {"error": err(e)}
out[name] = rec
return out
def main(path):
top = {"file": path}
try:
f = h5py.File(path, "r")
except Exception as e: # noqa: BLE001
top["open_error"] = err(e)
print(json.dumps(top))
return
objects = []
seen = set()
# Objects h5py cannot open have no ObjectID to deduplicate by; they are
# deduplicated by the address their hard link points at instead, as the
# probe deduplicates every object by header address.
seen_unopenable = set()
stack = [("/", None, None)]
while stack:
p, obj, link_addr = stack.pop()
if len(objects) >= MAX_OBJECTS:
top["truncated"] = True
break
rec = {"path": p}
try:
if obj is None:
obj = f[p]
key = hash(obj.id) # h5py ObjectID hash = (fileno, object address/token)
except Exception as e: # noqa: BLE001
if link_addr is not None:
if link_addr in seen_unopenable:
continue
seen_unopenable.add(link_addr)
rec["kind"] = "unknown"
rec["error"] = err(e)
objects.append(rec)
continue
if key in seen:
continue
seen.add(key)
if isinstance(obj, h5py.Dataset):
kind = "dataset"
elif isinstance(obj, h5py.Group):
kind = "group"
elif isinstance(obj, h5py.Datatype):
kind = "datatype"
else:
kind = "unknown"
rec["kind"] = kind
if kind == "dataset":
try:
dt = obj.dtype
rec["dtype"] = str(dt)
rec["shape"] = shape_of(obj.shape)
note_conversion(obj.id.get_type(), dt, rec)
if obj.shape is None:
hash_values(np.empty((0,), dtype=dt), dt, rec)
elif n_bytes(obj.shape, obj.id.get_type()) > MAX_BYTES:
rec["skipped"] = "too large"
else:
arr = np.empty(obj.shape, dtype=dt)
if arr.size:
try:
obj.read_direct(arr)
except Exception: # noqa: BLE001
arr = obj[()]
hash_values(arr, dt, rec)
except Exception as e: # noqa: BLE001
rec["error"] = err(e)
if kind != "datatype":
try:
rec["attrs"] = read_attrs(obj)
except Exception as e: # noqa: BLE001
rec["attrs_error"] = err(e)
if kind == "group":
try:
names = sorted(obj.keys(), key=lambda s: s.encode("utf-8", "surrogateescape"))
base = "" if p == "/" else p
kids = []
for n in names:
# The link's own type: `obj.get(n, getlink=True)` reports
# a user-defined link (type 64-255) as a HardLink.
try:
info = obj.id.links.get_info(n.encode("utf-8", "surrogateescape"))
except Exception: # noqa: BLE001
info = None
if info is not None and info.type != h5py.h5l.TYPE_HARD:
continue
addr = info.u if info is not None else None
kids.append((f"{base}/{n}", addr))
for k, addr in reversed(kids):
stack.append((k, None, addr))
except Exception as e: # noqa: BLE001
rec["list_error"] = err(e)
objects.append(rec)
top["objects"] = objects
print(json.dumps(top), flush=True)
# Exit without tearing down the h5py objects: freeing them for some files
# that hold references (hdf5's h5repack_attr_refs.h5, cve-2024-32623.h5)
# makes libhdf5 2.0 abort with "free(): chunks in smallbin corrupted"
# about half the time. That happens after the reading is done, so it says
# nothing about what h5py read, but it flipped those files between ok and
# h5py-cannot-read from one run to the next.
os._exit(0)
if __name__ == "__main__":
main(sys.argv[1])
conformance/report.py
+368
View File
@@ -0,0 +1,368 @@
#!/usr/bin/env python3
"""report.py <results_dir> <CONFORMANCE.md> <corpus_dir>
Render the sweep's results (compare.py's results.json plus the raw per-side
runs) as CONFORMANCE.md, and write <results_dir>/report-meta.json (commit,
date, versions) for check.py --update.
"""
import collections
import datetime
import json
import os
import platform
import subprocess
import sys
import h5py
import numpy
try:
import hdf5plugin
HDF5PLUGIN = hdf5plugin.version
except Exception: # noqa: BLE001
HDF5PLUGIN = "not installed"
R, OUT_MD, CORPUS = sys.argv[1], sys.argv[2], sys.argv[3]
HERE = os.path.dirname(os.path.abspath(__file__))
ROOT = os.path.dirname(HERE)
CLASSES = ["ok", "our-error", "mismatch", "h5py-cannot-read", "panic", "hang", "crash", "oom"]
def sh(*cmd, cwd=ROOT):
try:
return subprocess.run(cmd, cwd=cwd, capture_output=True, text=True, timeout=30).stdout.strip()
except Exception: # noqa: BLE001
return ""
def cpu_model():
try:
for ln in open("/proc/cpuinfo"):
if ln.startswith(("model name", "Model")):
return ln.split(":", 1)[1].strip()
except OSError:
pass
return platform.processor() or "unknown"
def mem_gib():
try:
for ln in open("/proc/meminfo"):
if ln.startswith("MemTotal:"):
return f"{int(ln.split()[1]) / 1048576:.0f} GiB"
except OSError:
pass
return "?"
res = json.load(open(os.path.join(R, "results.json")))
meta_run = json.load(open(os.path.join(R, "meta.json"))) if os.path.exists(os.path.join(R, "meta.json")) else {}
rows = res["rows"]
issues = res.get("issues", {})
# safe.directory: a checkout owned by another user (a container) is still ours to read
commit = sh("git", "-c", "safe.directory=*", "rev-parse", "HEAD") or os.environ.get("GITHUB_SHA", "unknown")
lib_dirty = sh("git", "-c", "safe.directory=*", "status", "--porcelain", "--", "crates", "Cargo.toml")
h5dump_v = sh("h5dump", "--version").replace("h5dump: ", "")
meta = {
"date": datetime.datetime.now(datetime.timezone.utc).strftime("%Y-%m-%d %H:%M UTC"),
"commit": commit + (" (library sources modified)" if lib_dirty else ""),
"reference": f"h5py {h5py.__version__} / HDF5 {h5py.version.hdf5_version}",
}
json.dump(meta, open(os.path.join(R, "report-meta.json"), "w"), indent=1)
pins = []
for ln in open(os.path.join(HERE, "corpus.txt")):
if ln.strip() and not ln.lstrip().startswith("#"):
name, url, rev, root, *_ = ln.split()
pins.append((name, url, rev, root))
by_corpus = collections.defaultdict(collections.Counter)
for r in rows:
by_corpus[r["corpus"]][r["class"]] += 1
total = collections.Counter(r["class"] for r in rows)
def ex_list(files, n=3):
s = ", ".join(f"`{f}`" for f in files[:n])
return s + (f" (+{len(files) - n} more)" if len(files) > n else "")
# --- known causes that are not clawhdf5 bugs --------------------------------
def is_h5py_be_vlen(i):
"""h5py returns the elements of a VL sequence of a big-endian base type
with their file (big-endian) bytes but a native-endian dtype."""
return (i["kind"] == "mismatch" and i["key"] in ("values", "attr-values")
and (i.get("ref_dtype") == "object") and (i.get("ours_dtype") or "").startswith("vlen(")
and ">" in (i.get("ours_dtype") or ""))
# Objects the reference (h5py 3.16 / HDF5 2.0) reads only because of an
# HDF5 2.0 bug, and that clawhdf5 refuses: each one reads past a buffer or
# returns bytes the file does not hold, and libhdf5's develop branch refuses all
# three. (file, object) -> why. Checked 2026-09-26 against HDF5 2.0.0
# and HDFGroup/hdf5 develop sources; see docs/known-issues.md.
LIBHDF5_BUGS = {
("cve_hdf5/cvefiles/cve-2025-2308.h5", "/Scale_offset_long_long_data_le"):
"scale-offset codes run past the end of the chunk: HDF5 2.0 reads past its buffer; "
"libhdf5's develop branch refuses the chunk (\"Buffer too short\")",
("cve_hdf5/cvefiles/cve-2025-44904.h5", "/Scale_offset_float_data_le"):
"unfiltered chunks of 38 and 37 bytes for 48-byte chunks: HDF5 2.0 fills the rest with "
"whatever its buffer held; libhdf5's develop branch refuses them (\"incorrect chunk size returned "
"from index for unfiltered chunk\")",
("hdf5/test/testfiles/bad_nbit_parms_walk.h5", "/Nbit_int_data_le"):
"an N-Bit parameter list one value short: HDF5 2.0 reads past the list; libhdf5's own "
"test (`test_filter_bad_params`, test/dsets.c) now requires the read to fail",
}
def is_libhdf5_bug(rel, i):
return i["kind"] == "our-error" and any(
f == rel and i["detail"].startswith(obj + ":") for (f, obj) in LIBHDF5_BUGS)
known = collections.defaultdict(list)
for r in rows:
iss = issues.get(r["file"], [])
if r["class"] == "mismatch" and iss and all(is_h5py_be_vlen(i) for i in iss):
known["h5py-be-vlen"].append(r["file"])
if r["class"] == "our-error" and iss and all(is_libhdf5_bug(r["file"], i) for i in iss):
known["libhdf5-2.0"].append(r["file"])
# --- the CVE corpus: clawhdf5 vs h5dump vs h5py ------------------------------
def side(run, name):
p = os.path.join(R, "runs", run, name)
if not os.path.exists(p + ".rc"):
return None
rc = int(open(p + ".rc").read().strip() or -1)
err = open(p + ".err", errors="replace").read()
try:
j = json.load(open(p + ".json"))
except Exception: # noqa: BLE001
j = None
return rc, err, j
def outcome(s, rust=False):
"""-> (bucket, text). bucket in read / error / panic / crash / hang / oom."""
if s is None:
return "missing", "not run"
rc, err, j = s
if rc in (137, 124):
return "hang", "hang (killed at timeout)"
if "memory allocation of" in err or "MemoryError" in err or "bad_alloc" in err or "Cannot allocate" in err:
return "oom", "out of memory"
if rust and (rc == 101 or "PANIC:" in err):
return "panic", "panic"
if "overflowed its stack" in err:
return "crash", "stack overflow"
if rc == 139:
return "crash", "SIGSEGV"
if rc == 134:
return "crash", "SIGABRT" + (" (heap corruption)" if ("corrupted" in err or "free()" in err) else "")
if rc > 128:
return "crash", f"signal {rc - 128}"
if j is None:
return ("error", "error exit") if rc in (0, 1) else ("crash", f"exit {rc}")
if "open_error" in j:
return "error", "open error"
objs = j.get("objects", [])
ne = sum(1 for o in objs for k in ("error", "attrs_error", "list_error") if k in o)
ne += sum(1 for o in objs for a in (o.get("attrs") or {}).values() if "error" in a)
return "read", f"read {len(objs)} obj" + (f", {ne} errors" if ne else "")
def h5dump_outcome(s):
if s is None:
return "missing", "not run"
rc, err, _ = s
if rc in (137, 124):
return "hang", "hang (killed at timeout)"
if "memory allocation" in err or "Cannot allocate" in err:
return "oom", "out of memory"
if rc == 139:
return "crash", "SIGSEGV"
if rc == 134:
return "crash", "SIGABRT" + (" (heap corruption)" if ("corrupted" in err or "free()" in err) else "")
if rc > 128:
return "crash", f"signal {rc - 128}"
return ("read", "ok") if rc == 0 else ("error", "error exit")
cve_rows = []
buckets = {"clawhdf5": collections.Counter(), "h5dump": collections.Counter(), "h5py": collections.Counter()}
ours_panic = {r["file"] for r in rows if r["class"] == "panic"}
for r in rows:
if r["corpus"] != "cve_hdf5":
continue
run = r["file"].replace("/", "__")
o = outcome(side(run, "ours"), rust=True)
if o[0] == "read" and r["file"] in ours_panic:
o = ("panic", "caught panic")
p = outcome(side(run, "ref"))
d = h5dump_outcome(side(run, "h5dump"))
buckets["clawhdf5"][o[0]] += 1
buckets["h5py"][p[0]] += 1
buckets["h5dump"][d[0]] += 1
cve_rows.append((r["file"].split("/", 1)[1], d[1], p[1], o[1], r["class"]))
# --- render -----------------------------------------------------------------
L = []
w = L.append
w("# clawhdf5 conformance report")
w("")
w("Every HDF5 file of eight public corpora (pinned by commit) is read twice — by")
w("clawhdf5 (`conformance/probe`, the same `clawhdf5-format` calls the facade")
w("makes) and by h5py/libhdf5 (`conformance/ref.py`) — and the two readings are")
w("compared object by object: the set of hard-linked objects, each dataset's and")
w("attribute's shape, and a SHA-256 of its values in a canonical encoding. The")
w("CVE corpus is also run through `h5dump`. Each side runs under a timeout and an")
w("address-space limit, so a hang, crash or runaway allocation is recorded, not")
w("fatal. This file is generated by `conformance/run.sh`; do not edit it by hand.")
w("")
w("## Run")
w("")
w("| | |")
w("|---|---|")
w(f"| date | {meta['date']} |")
w(f"| clawhdf5 commit | `{meta['commit']}` |")
w(f"| machine | `{platform.node()}`: {cpu_model()}, {os.cpu_count()} CPUs, {mem_gib()}, {platform.system()} {platform.release()} {platform.machine()} |")
w(f"| command | `{os.environ.get('CONFORMANCE_CMD', 'conformance/run.sh')}` |")
w(f"| rustc | {sh('rustc', '-V')} |")
w(f"| reference | h5py {h5py.__version__}, HDF5 {h5py.version.hdf5_version}, numpy {numpy.__version__}, hdf5plugin {HDF5PLUGIN}, Python {platform.python_version()} |")
w(f"| h5dump | {h5dump_v} (CVE corpus only) |")
if meta_run:
w(f"| limits | {meta_run.get('timeout_s')} s timeout (SIGKILL), {int(meta_run.get('mem_kb', 0)) // 1024} MiB address space, per process; {meta_run.get('jobs')} files in parallel |")
w(f"| runtime | {meta_run.get('probe_seconds')} s probing + comparing ({meta_run.get('build_seconds')} s fetch/build before it) |")
w("")
w("## Results")
w("")
w("A file's class is the first that applies:")
w("")
w("- **panic / hang / crash / oom** — clawhdf5 panicked (caught per object or not), hit the timeout, died on a signal, or failed an allocation. The CI gate fails on any of these.")
w("- **h5py-cannot-read** — libhdf5 could not open the file (or itself crashed or hung). Nothing to compare against; most are the deliberately malformed CVE reproducers.")
w("- **our-error** — clawhdf5 returned an error for something h5py reads.")
w("- **mismatch** — both read it, but the shapes, values, object set or attribute set differ.")
w("- **ok** — every object h5py reads, clawhdf5 reads identically.")
w("")
w("| corpus | files | " + " | ".join(CLASSES) + " |")
w("|---" * (len(CLASSES) + 2) + "|")
for c in sorted(by_corpus):
cnt = by_corpus[c]
w(f"| {c} | {sum(cnt.values())} | " + " | ".join(str(cnt.get(k, 0)) for k in CLASSES) + " |")
w(f"| **all** | **{len(rows)}** | " + " | ".join(f"**{total.get(k, 0)}**" for k in CLASSES) + " |")
w("")
if known["h5py-be-vlen"]:
w(f"{len(known['h5py-be-vlen'])} of the {total.get('mismatch', 0)} mismatches are a known h5py bug, "
"not ours (see *Known not-our-bug*).")
w("")
if known["libhdf5-2.0"]:
w(f"{len(known['libhdf5-2.0'])} of the {total.get('our-error', 0)} our-errors are corrupt data that "
"HDF5 2.0 reads only through a bug and clawhdf5 refuses (see *Known not-our-bug*).")
w("")
w("Corpora (fetched by `conformance/fetch-corpus.sh` into the gitignored `conformance/.cache/`):")
w("")
w("| corpus | source | commit |")
w("|---|---|---|")
for name, url, rev, root in pins:
w(f"| {name} | {url.removesuffix('.git')}" + ("" if root == "." else f" (`{root}`)") + f" | `{rev[:12]}` |")
w("")
w("## Panics, hangs, crashes, out-of-memory")
w("")
if not res["panics"]:
w("None.")
else:
for p in res["panics"]:
w(f"- `{p['file']}` [{p['class']}] {p['detail']}")
w("")
w("## Our-error root causes")
w("")
w("Grouped by normalised error message. *files* counts files whose class this cause affects.")
w("")
w("| files | objects | error | examples |")
w("|---:|---:|---|---|")
for k, v in res["root_causes"].items():
w(f"| {v['files']} | {v['count']} | `{k.replace('|', '/')}` | {ex_list(v['file_list'])} |")
w("")
w("## Mismatch root causes")
w("")
w("| files | objects | cause | examples |")
w("|---:|---:|---|---|")
for k, v in res["mismatch_causes"].items():
w(f"| {v['files']} | {v['count']} | `{k.replace('|', '/')}` | {ex_list(v['file_list'])} |")
w("")
w("## CVE corpus: clawhdf5 vs h5dump vs h5py")
w("")
w(f"The {len(cve_rows)} files of [HDFGroup/cve_hdf5](https://github.com/HDFGroup/cve_hdf5) — reproducers for")
w("published libhdf5 CVEs and fuzzer finds. *read* = produced output (possibly with per-object")
w("errors), *error* = refused cleanly. h5dump exits non-zero on any error anywhere in a file, so")
w("its read/error split is not comparable with the other two rows; the panic, crash, hang and oom")
w("columns are.")
w("")
w("| tool | read | error | panic | crash | hang | oom |")
w("|---|---:|---:|---:|---:|---:|---:|")
for tool, label in (("clawhdf5", "clawhdf5"), ("h5dump", f"h5dump {h5dump_v.split()[-1] if h5dump_v else ''}"),
("h5py", f"h5py {h5py.__version__} / HDF5 {h5py.version.hdf5_version}")):
b = buckets[tool]
w(f"| {label} | " + " | ".join(str(b.get(k, 0)) for k in ("read", "error", "panic", "crash", "hang", "oom")) + " |")
w("")
w("<details><summary>Per-file outcomes</summary>")
w("")
w("| file | h5dump | h5py | clawhdf5 | class |")
w("|---|---|---|---|---|")
for f, d, p, o, cls in cve_rows:
w(f"| {f} | {d} | {p} | {o} | {cls} |")
w("")
w("</details>")
w("")
w("## Known not-our-bug")
w("")
w("- **h5py big-endian variable-length sequences.** h5py returns the elements of a VL sequence")
w(" whose base type is big-endian with the file's big-endian bytes but a native (little-endian)")
w(" numpy dtype, so the values it reports are byte-swapped garbage; `h5dump` prints the values")
w(" clawhdf5 reads. Reproducer: `h5py.vlen_dtype(np.dtype('>f4'))` dataset holding `[1.0, 2.0]`")
w(" reads back in h5py as `[4.6e-41, 9.0e-44]`. Affected here: "
+ (ex_list(sorted(known["h5py-be-vlen"]), 10) if known["h5py-be-vlen"] else "none") + ".")
w("- **Non-IEEE floats and partial-precision integers (N-Bit).** libhdf5 converts a float whose")
w(" bit layout is not IEEE (e.g. `H5Tset_precision` for the N-Bit filter) or an integer with a")
w(" bit offset / reduced precision into the plain numpy type of the same size. The probe")
w(" compares such values as converted numbers, not raw file bytes (before 2026-09-25 it compared")
w(" raw bytes, which reported every N-Bit float dataset as a mismatch).")
if res["incomparable"]:
w("- **Types h5py widens.** Where h5py reads a type into a numpy type of a different size")
w(" (FP8 -> float16, bfloat16 -> float32, x87 long double -> float128) the values are not")
w(" compared (shape and presence still are): "
+ ", ".join(f"{k} ({n}x)" for k, n in res["incomparable"]) + ".")
w("- **Corrupt data HDF5 2.0 reads through a bug.** clawhdf5 refuses these objects; h5py 3.16 /")
w(" HDF5 2.0 returns values for them that the file does not hold:")
for (f, obj), why in sorted(LIBHDF5_BUGS.items()):
here = "" if f in known["libhdf5-2.0"] else " (not an our-error in this run)"
w(f" - `{f}` `{obj}`: {why}{here}.")
w("- **References** are compared by presence only (`R`), not by target.")
w("")
if res.get("ref_only_errors"):
w("## Objects h5py fails on but clawhdf5 reads")
w("")
for k, n in res["ref_only_errors"][:15]:
w(f"- {n} x `{k}`")
w("")
w("## Reproduce")
w("")
w("```sh")
w("# needs: Rust, python3 with h5py numpy hdf5plugin (conformance/requirements.txt), h5dump (hdf5-tools), git")
w("CLAWHDF5_PYTHON=/path/to/venv/bin/python conformance/run.sh")
w("```")
w("")
w("The corpus (about 450 MB of sparse checkouts) is cached in `conformance/.cache/`; results for")
w("every file, both sides' raw JSON and stderr, are in `conformance/.cache/results/`.")
w("`conformance/baseline.json` holds the ok files the nightly CI job (`.gitea/workflows/conformance.yml`)")
w("must keep; `conformance/run.sh --update-baseline` rewrites it.")
with open(OUT_MD, "w") as fh:
fh.write("\n".join(L) + "\n")
conformance/requirements.txt
+6
View File
@@ -0,0 +1,6 @@
# The reference side of the conformance sweep. Pinned so the nightly job and a
# local run compare against the same libhdf5 (h5py wheels bundle it).
h5py==3.16.0
numpy==2.5.3
hdf5plugin==7.1.0
netCDF4==1.7.4
conformance/run.sh
Executable
+88
View File
@@ -0,0 +1,88 @@
#!/usr/bin/env bash
# conformance/run.sh — the clawhdf5 conformance sweep, end to end.
#
# fetch the pinned corpora (cached) -> build the probe -> probe every file
# with clawhdf5 and with h5py (and h5dump for the CVE corpus), each under a
# timeout and a memory limit -> compare -> write CONFORMANCE.md -> check the
# result against conformance/baseline.json.
#
# Usage: conformance/run.sh [--no-fetch] [--no-report] [--update-baseline]
#
# Environment:
# CLAWHDF5_PYTHON python with h5py, numpy, hdf5plugin (default: repo .venv, then python3)
# CONFORMANCE_CACHE corpus / build / results cache (default: conformance/.cache)
# CONFORMANCE_OUT results directory (default: $CONFORMANCE_CACHE/results)
# CONFORMANCE_REPORT report path (default: CONFORMANCE.md at the repo root)
# JOBS parallel files (default: nproc)
# CONFORMANCE_PROBE use this prebuilt probe binary instead of building one
# TMO / MEM_KB per-process timeout in seconds (20) / address-space limit in KiB (4 GiB)
#
# Exit status: 0 = gate passed; 1 = a panic/hang/crash/oom in clawhdf5, or the
# ok count fell below the baseline, or a baseline-ok file regressed; 2 = setup error.
set -euo pipefail
HERE="$(cd "$(dirname "$0")" && pwd)"
ROOT="$(cd "$HERE/.." && pwd)"
FETCH=1 REPORT=1 UPDATE=0
for a in "$@"; do
case "$a" in
--no-fetch) FETCH=0 ;;
--no-report) REPORT=0 ;;
--update-baseline) UPDATE=1 ;;
-h|--help) sed -n '2,23p' "$0"; exit 0 ;;
*) echo "unknown argument: $a" >&2; exit 2 ;;
esac
done
export PATH="$HOME/.cargo/bin:$PATH"
CACHE="${CONFORMANCE_CACHE:-$HERE/.cache}"
mkdir -p "$CACHE"; CACHE="$(cd "$CACHE" && pwd)"
OUT="${CONFORMANCE_OUT:-$CACHE/results}"
REPORT_PATH="${CONFORMANCE_REPORT:-$ROOT/CONFORMANCE.md}"
JOBS="${JOBS:-$(nproc 2>/dev/null || echo 4)}"
if [ -n "${CLAWHDF5_PYTHON:-}" ]; then PY="$CLAWHDF5_PYTHON"
elif [ -x "$ROOT/.venv/bin/python" ]; then PY="$ROOT/.venv/bin/python"
else PY="$(command -v python3)"; fi
export PY TMO="${TMO:-20}" MEM_KB="${MEM_KB:-4194304}"
command -v h5dump >/dev/null || { echo "error: h5dump not found (install hdf5-tools)" >&2; exit 2; }
"$PY" -c 'import h5py, numpy, hdf5plugin' || { echo "error: $PY lacks h5py/numpy/hdf5plugin" >&2; exit 2; }
t0=$(date +%s)
[ "$FETCH" = 1 ] && bash "$HERE/fetch-corpus.sh" "$CACHE"
C="$CACHE/corpus"
[ -d "$C" ] || { echo "error: no corpus in $C (run without --no-fetch)" >&2; exit 2; }
if [ -n "${CONFORMANCE_PROBE:-}" ]; then
export PROBE="$CONFORMANCE_PROBE" # a prebuilt probe, e.g. an older one for a before/after
else
echo "== building the probe"
CARGO_TARGET_DIR="${CARGO_TARGET_DIR:-$CACHE/target}" \
cargo build -q --release --manifest-path "$HERE/probe/Cargo.toml"
export PROBE="${CARGO_TARGET_DIR:-$CACHE/target}/release/conformance-probe"
fi
t1=$(date +%s)
rm -rf "$OUT"; mkdir -p "$OUT"
"$PY" "$HERE/list_files.py" "$C" > "$OUT/files.txt"
echo "== probing $(wc -l <"$OUT/files.txt") files, $JOBS at a time (timeout ${TMO}s, limit $((MEM_KB / 1024)) MiB)"
export C OUT HERE
# The shell's "Segmentation fault (core dumped)" notices go to probe.log; the
# signals themselves are recorded in each side's .rc.
xargs -a "$OUT/files.txt" -d '\n' -P "$JOBS" -I{} bash -c '
f="$1"; d="$OUT/runs/${f//\//__}"
case "$f" in cve_hdf5/*) export WITH_H5DUMP=1 ;; esac
"$HERE/run_one.sh" "$C/$f" "$d"' _ {} 2>"$OUT/probe.log"
echo "== comparing"
"$PY" "$HERE/compare.py" "$OUT" >/dev/null
t2=$(date +%s)
cat > "$OUT/meta.json" <<EOF
{"build_seconds": $((t1 - t0)), "probe_seconds": $((t2 - t1)), "jobs": $JOBS, "timeout_s": $TMO, "mem_kb": $MEM_KB}
EOF
export CONFORMANCE_CMD="${CONFORMANCE_CMD:-conformance/run.sh${*:+ $*}}"
if [ "$REPORT" = 1 ]; then
"$PY" "$HERE/report.py" "$OUT" "$REPORT_PATH" "$C"
echo "== wrote $REPORT_PATH"
fi
if [ "$UPDATE" = 1 ]; then
"$PY" "$HERE/check.py" "$OUT" "$HERE/baseline.json" --update
fi
"$PY" "$HERE/check.py" "$OUT" "$HERE/baseline.json"
conformance/run_one.sh
Executable
+27
View File
@@ -0,0 +1,27 @@
#!/usr/bin/env bash
# run_one.sh <file> <outdir>
#
# Probe one file with clawhdf5 (PROBE) and with h5py (PY ref.py), and with
# h5dump too when WITH_H5DUMP is set. Each side runs under a timeout (TMO
# seconds, SIGKILL) and an address-space limit (MEM_KB), with core dumps off.
# Writes <outdir>/<side>.{json,err,rc}; rc 137 = killed by the timeout.
set -u
f="$1"; out="$2"; mkdir -p "$out"
HERE="$(cd "$(dirname "$0")" && pwd)"
: "${PROBE:?PROBE must name the conformance-probe binary}"
: "${PY:?PY must name a python with h5py}"
TMO="${TMO:-20}"
MEM_KB="${MEM_KB:-4194304}"
run() { # name cmd...
local name=$1; shift
( ulimit -v "$MEM_KB"; ulimit -c 0; RUST_BACKTRACE=1 exec timeout -s KILL "$TMO" "$@" ) \
>"$out/$name.json" 2>"$out/$name.err"
echo $? >"$out/$name.rc"
}
run ours "$PROBE" "$f"
run ref "$PY" "$HERE/ref.py" "$f"
if [ -n "${WITH_H5DUMP:-}" ]; then
run h5dump h5dump "$f"
: >"$out/h5dump.json" # h5dump's text dump is not compared, only its exit status
fi
exit 0
crates/clawhdf5-accel/Cargo.toml
+2 -1
View File
@@ -1,7 +1,8 @@
[package]
name = "clawhdf5-accel"
version = "2.4.0"
version = "2.7.0"
edition = "2024"
rust-version.workspace = true
description = "SIMD-accelerated operations for rustyhdf5"
license = "MIT"
repository = "https://git.redclaw.dev/quantumclaw/clawhdf5"
crates/clawhdf5-accel/src/avx2.rs
+49
View File
@@ -25,6 +25,55 @@ unsafe fn hsum_256(v: __m256) -> f32 {
_mm_cvtss_f32(result)
}
/// AVX2 dot product of two `i8` slices, widened to `i32`.
///
/// Each 16-byte half is sign-extended to sixteen `i16` lanes and multiplied
/// pairwise with `madd_epi16`, which sums adjacent products straight into
/// eight `i32` lanes — the widening that an autovectorised scalar loop does
/// in several shuffles is one instruction here. A pair sum is at most
/// `2 * 127 * 127`, far inside `i32`.
///
/// # Safety
/// Caller must verify is_x86_feature_detected!("avx2").
// SAFETY: Caller must have verified AVX2 via is_x86_feature_detected!.
#[target_feature(enable = "avx2")]
pub unsafe fn dot_i8(a: &[i8], b: &[i8]) -> i32 {
// SAFETY: Caller guarantees AVX2 is available per the # Safety contract;
// every load reads 32 bytes at an index checked against `len` first.
unsafe {
assert_eq!(a.len(), b.len());
let len = a.len();
let mut i = 0;
let mut acc0 = _mm256_setzero_si256();
let mut acc1 = _mm256_setzero_si256();
while i + 32 <= len {
let va = _mm256_loadu_si256(a.as_ptr().add(i).cast());
let vb = _mm256_loadu_si256(b.as_ptr().add(i).cast());
let a_lo = _mm256_cvtepi8_epi16(_mm256_castsi256_si128(va));
let b_lo = _mm256_cvtepi8_epi16(_mm256_castsi256_si128(vb));
let a_hi = _mm256_cvtepi8_epi16(_mm256_extracti128_si256(va, 1));
let b_hi = _mm256_cvtepi8_epi16(_mm256_extracti128_si256(vb, 1));
acc0 = _mm256_add_epi32(acc0, _mm256_madd_epi16(a_lo, b_lo));
acc1 = _mm256_add_epi32(acc1, _mm256_madd_epi16(a_hi, b_hi));
i += 32;
}
// Horizontal sum of the eight i32 lanes.
let v = _mm256_add_epi32(acc0, acc1);
let s128 = _mm_add_epi32(_mm256_castsi256_si128(v), _mm256_extracti128_si256(v, 1));
let s64 = _mm_add_epi32(s128, _mm_unpackhi_epi64(s128, s128));
let s32 = _mm_add_epi32(s64, _mm_shuffle_epi32(s64, 0b01));
let mut sum = _mm_cvtsi128_si32(s32);
while i < len {
sum += i32::from(a[i]) * i32::from(b[i]);
i += 1;
}
sum
}
}
/// AVX2 dot product for f32 slices.
///
/// # Safety
crates/clawhdf5-accel/src/lib.rs
+109 -1
View File
@@ -122,6 +122,36 @@ pub fn dot_product(a: &[f32], b: &[f32]) -> f32 {
}
}
/// Dot product of two `i8` slices, widened to `i32`.
///
/// The kernel behind int8-quantised vector search. On x86-64 it uses the AVX2
/// path whenever AVX2 is present (including on AVX-512 machines, where it is
/// what the f32 kernels use too on a default build). On aarch64 it uses the
/// ARMv8.2 `SDOT` instruction when the CPU has the dot-product extension, and
/// plain NEON otherwise.
pub fn dot_i8(a: &[i8], b: &[i8]) -> i32 {
match detect_backend() {
#[cfg(target_arch = "aarch64")]
Backend::Neon => {
if std::arch::is_aarch64_feature_detected!("dotprod") {
// SAFETY: the dotprod extension was just detected at runtime.
unsafe { neon::dot_i8_dotprod(a, b) }
} else {
// SAFETY: NEON is always available on aarch64.
unsafe { neon::dot_i8(a, b) }
}
}
#[cfg(target_arch = "x86_64")]
// SAFETY: both variants imply AVX2 was detected at runtime (the
// AVX-512 backend is only selected on CPUs that also have AVX2).
Backend::Avx2 | Backend::Avx512 if is_x86_feature_detected!("avx2") => unsafe {
avx2::dot_i8(a, b)
},
_ => scalar::dot_i8(a, b),
}
}
/// Compute the L2 norm (magnitude) of a vector.
pub fn vector_norm(v: &[f32]) -> f32 {
dot_product(v, v).sqrt()
@@ -207,7 +237,10 @@ pub fn f16_to_f32_batch(input: &[u16], output: &mut [f32]) {
convert::f16_to_f32_batch(input, output);
}
/// Compute Fletcher-32 checksum.
/// Compute a textbook Fletcher-32 checksum (both sums start at 0xffff).
///
/// This is not HDF5's checksum; the Fletcher-32 I/O filter uses
/// `clawhdf5_format::checksum::fletcher32`.
pub fn checksum_fletcher32(data: &[u8]) -> u32 {
checksum::checksum_fletcher32(data)
}
@@ -713,3 +746,78 @@ mod tests {
}
}
}
#[cfg(test)]
mod dot_i8_tests {
use super::*;
fn codes(n: usize, seed: u64) -> Vec<i8> {
let mut state = seed;
(0..n)
.map(|_| {
state = state
.wrapping_mul(6_364_136_223_846_793_005)
.wrapping_add(1_442_695_040_888_963_407);
// Full range, including the extremes.
((state >> 56) as u8) as i8
})
.collect()
}
#[test]
fn dispatched_kernel_matches_scalar_exactly() {
// Integer arithmetic: the SIMD path must agree bit for bit, at every
// length — including ones that are not multiples of the 32-byte block,
// which exercise the tail.
for len in [0, 1, 7, 31, 32, 33, 63, 64, 100, 384, 385, 1536] {
let a = codes(len, 1 + len as u64);
let b = codes(len, 1000 + len as u64);
assert_eq!(dot_i8(&a, &b), scalar::dot_i8(&a, &b), "len {len}");
}
}
/// Dispatch only ever takes one path on a given CPU, so on a machine with
/// the dot-product extension the plain-NEON kernel would otherwise go
/// untested. Check each aarch64 kernel against scalar directly.
#[cfg(target_arch = "aarch64")]
#[test]
fn every_aarch64_kernel_matches_scalar_exactly() {
for len in [0, 1, 7, 15, 16, 17, 31, 32, 33, 63, 64, 100, 384, 385, 1536] {
let a = codes(len, 7 + len as u64);
let b = codes(len, 7000 + len as u64);
let want = scalar::dot_i8(&a, &b);
// SAFETY: NEON is always available on aarch64.
assert_eq!(unsafe { neon::dot_i8(&a, &b) }, want, "neon, len {len}");
if std::arch::is_aarch64_feature_detected!("dotprod") {
// SAFETY: the dotprod extension was just detected.
assert_eq!(
unsafe { neon::dot_i8_dotprod(&a, &b) },
want,
"dotprod, len {len}"
);
}
}
// The extremes, through both kernels.
let lo = vec![-128i8; 4096];
let hi = vec![127i8; 4096];
// SAFETY: NEON is always available on aarch64.
assert_eq!(unsafe { neon::dot_i8(&lo, &lo) }, 4096 * 128 * 128);
// SAFETY: NEON is always available on aarch64.
assert_eq!(unsafe { neon::dot_i8(&lo, &hi) }, -4096 * 128 * 127);
if std::arch::is_aarch64_feature_detected!("dotprod") {
// SAFETY: the dotprod extension was just detected.
assert_eq!(unsafe { neon::dot_i8_dotprod(&lo, &lo) }, 4096 * 128 * 128);
// SAFETY: the dotprod extension was just detected.
assert_eq!(unsafe { neon::dot_i8_dotprod(&lo, &hi) }, -4096 * 128 * 127);
}
}
#[test]
fn extremes_do_not_overflow() {
// -128 * -128 is the largest product; a long run of it must still fit.
let a = vec![-128i8; 4096];
assert_eq!(dot_i8(&a, &a), 4096 * 128 * 128);
let b = vec![127i8; 4096];
assert_eq!(dot_i8(&a, &b), -4096 * 128 * 127);
}
}
crates/clawhdf5-accel/src/neon.rs
+127
View File
@@ -180,3 +180,130 @@ pub fn checksum_fletcher32(data: &[u8]) -> u32 {
(sum2 << 16) | sum1
}
/// NEON dot product of two `i8` slices, widened to `i32`, for any aarch64 CPU.
///
/// `vmull_s8` multiplies eight lanes into `i16` — even `-128 * -128` is 16 384,
/// inside `i16` — and `vpadalq_s16` adds adjacent pairs of those into `i32`
/// accumulators, so nothing can overflow before the final horizontal sum.
///
/// CPUs with the ARMv8.2 dot-product extension should use
/// [`dot_i8_dotprod`], which does the multiply and the accumulate in one
/// instruction.
///
/// # Safety
/// Caller must ensure aarch64 target (NEON always available).
// SAFETY: NEON is always available on aarch64 targets; caller guarantees aarch64.
#[target_feature(enable = "neon")]
pub unsafe fn dot_i8(a: &[i8], b: &[i8]) -> i32 {
assert_eq!(a.len(), b.len());
let len = a.len();
let mut i = 0;
let mut acc0 = vdupq_n_s32(0);
let mut acc1 = vdupq_n_s32(0);
while i + 16 <= len {
// SAFETY: NEON is available per the # Safety contract, and both
// 16-byte loads start at an index checked against `len` above.
unsafe {
let va = vld1q_s8(a.as_ptr().add(i));
let vb = vld1q_s8(b.as_ptr().add(i));
acc0 = vpadalq_s16(acc0, vmull_s8(vget_low_s8(va), vget_low_s8(vb)));
acc1 = vpadalq_s16(acc1, vmull_high_s8(va, vb));
}
i += 16;
}
let mut sum = vaddvq_s32(vaddq_s32(acc0, acc1));
while i < len {
sum += i32::from(a[i]) * i32::from(b[i]);
i += 1;
}
sum
}
/// One `SDOT`: for each of the four `i32` lanes of `acc`, add the dot
/// product of the corresponding four `i8` pairs from `a` and `b`.
///
/// Written as inline assembly because the `vdotq_s32` intrinsic is still
/// behind the unstable `stdarch_neon_dotprod` feature; inline assembly is
/// stable on aarch64.
///
/// # Safety
/// Caller must ensure the CPU supports the `dotprod` extension.
#[inline]
#[target_feature(enable = "neon,dotprod")]
unsafe fn sdot(acc: int32x4_t, a: int8x16_t, b: int8x16_t) -> int32x4_t {
let mut acc = acc;
// SAFETY: `dotprod` is enabled for this function and the caller
// guarantees the CPU supports it. The instruction reads only its three
// vector registers and touches no memory.
unsafe {
std::arch::asm!(
"sdot {acc:v}.4s, {a:v}.16b, {b:v}.16b",
acc = inout(vreg) acc,
a = in(vreg) a,
b = in(vreg) b,
options(pure, nomem, nostack),
);
}
acc
}
/// NEON dot product of two `i8` slices using the ARMv8.2 dot-product
/// extension (`SDOT`): sixteen multiply-accumulates per instruction, straight
/// into `i32` lanes.
///
/// Present on the cores this crate actually runs on — Cortex-A76 and later
/// (Raspberry Pi 5, current Android phones), Neoverse-N1 (Graviton2, Ampere
/// Altra), and every Apple Silicon generation.
///
/// # Safety
/// Caller must verify `is_aarch64_feature_detected!("dotprod")`.
// SAFETY: caller has verified the dotprod extension at runtime.
#[target_feature(enable = "neon,dotprod")]
pub unsafe fn dot_i8_dotprod(a: &[i8], b: &[i8]) -> i32 {
assert_eq!(a.len(), b.len());
let len = a.len();
let mut i = 0;
let mut acc0 = vdupq_n_s32(0);
let mut acc1 = vdupq_n_s32(0);
// Two independent accumulators so consecutive SDOTs are not serialised on
// one register.
while i + 32 <= len {
// SAFETY: dotprod is available per the # Safety contract, and every
// 16-byte load starts at an index checked against `len` above.
unsafe {
acc0 = sdot(
acc0,
vld1q_s8(a.as_ptr().add(i)),
vld1q_s8(b.as_ptr().add(i)),
);
acc1 = sdot(
acc1,
vld1q_s8(a.as_ptr().add(i + 16)),
vld1q_s8(b.as_ptr().add(i + 16)),
);
}
i += 32;
}
if i + 16 <= len {
// SAFETY: as above; the load is bounds-checked by this condition.
unsafe {
acc0 = sdot(
acc0,
vld1q_s8(a.as_ptr().add(i)),
vld1q_s8(b.as_ptr().add(i)),
);
}
i += 16;
}
let mut sum = vaddvq_s32(vaddq_s32(acc0, acc1));
while i < len {
sum += i32::from(a[i]) * i32::from(b[i]);
i += 1;
}
sum
}
crates/clawhdf5-accel/src/scalar.rs
+30
View File
@@ -140,3 +140,33 @@ fn f16_to_f32_soft(h: u16) -> f32 {
f32::from_bits(f32_bits)
}
/// Dot product of two `i8` slices, widened to `i32`.
///
/// `dim` terms of at most `127 * 127` fit an `i32` for any realistic
/// dimension (over 130 000 terms before overflow is possible).
pub fn dot_i8(a: &[i8], b: &[i8]) -> i32 {
assert_eq!(a.len(), b.len());
// Four independent accumulators over 32-lane blocks: the widening product
// has to sit in a fixed-length chunk for the vectoriser to see it, and the
// separate accumulators keep it off one dependency chain.
const LANE: usize = 8;
let (a_blocks, a_tail) = a.as_chunks::<{ LANE * 4 }>();
let (b_blocks, b_tail) = b.as_chunks::<{ LANE * 4 }>();
let mut acc = [0i32; 4];
for (x, y) in a_blocks.iter().zip(b_blocks) {
for (lane, slot) in acc.iter_mut().enumerate() {
let mut sum = 0i32;
for k in 0..LANE {
sum += i32::from(x[lane * LANE + k]) * i32::from(y[lane * LANE + k]);
}
*slot += sum;
}
}
let tail: i32 = a_tail
.iter()
.zip(b_tail)
.map(|(&x, &y)| i32::from(x) * i32::from(y))
.sum();
acc[0] + acc[1] + acc[2] + acc[3] + tail
}
crates/clawhdf5-agent/Cargo.toml
+20 -10
View File
@@ -1,7 +1,8 @@
[package]
name = "clawhdf5-agent"
version = "2.4.0"
version = "2.7.0"
edition = "2024"
rust-version.workspace = true
description = "HDF5-backed persistent memory store for on-device AI agents"
license = "MIT"
repository = "https://git.redclaw.dev/quantumclaw/clawhdf5"
@@ -10,14 +11,18 @@ keywords = ["agent", "memory", "hdf5", "vector-search", "embedding"]
categories = ["database", "science", "algorithms"]
[dependencies]
clawhdf5-format = { path = "../clawhdf5-format", version = "2.4.0", features = ["parallel", "fast-checksum"] }
clawhdf5 = { path = "../clawhdf5", version = "2.4.0" }
clawhdf5-io = { path = "../clawhdf5-io", version = "2.4.0", features = ["mmap"] }
clawhdf5-accel = { path = "../clawhdf5-accel", version = "2.4.0" }
clawhdf5-ann = { path = "../clawhdf5-ann", version = "2.4.0", optional = true }
clawhdf5-gpu = { path = "../clawhdf5-gpu", version = "2.4.0", optional = true, default-features = false }
clawhdf5-format = { path = "../clawhdf5-format", version = "2.7.0", features = ["parallel", "fast-checksum"] }
clawhdf5 = { path = "../clawhdf5", version = "2.7.0" }
clawhdf5-io = { path = "../clawhdf5-io", version = "2.7.0", features = ["mmap"] }
clawhdf5-accel = { path = "../clawhdf5-accel", version = "2.7.0" }
clawhdf5-ann = { path = "../clawhdf5-ann", version = "2.7.0", optional = true }
clawhdf5-gpu = { path = "../clawhdf5-gpu", version = "2.7.0", optional = true, default-features = false }
serde = { workspace = true }
byteorder = "1"
# Signed checkpoints (MemoryConfig-independent; see `signing`). Pure Rust.
ed25519-dalek = { version = "2", features = ["rand_core"] }
sha2 = "0.10"
rand_core = { version = "0.6", features = ["getrandom"] }
half = { workspace = true, optional = true }
rayon = { version = "1", optional = true }
matrixmultiply = { version = "0.3", optional = true }
@@ -44,10 +49,16 @@ harness = false
name = "memory_bench"
harness = false
[[bench]]
name = "multimodal_bench"
harness = false
[features]
default = ["float16", "hnsw"]
default = ["float16", "hnsw", "parallel"]
float16 = ["half"]
parallel = ["rayon"]
# Rayon-parallel brute-force search strategies, and a parallel bulk build of
# the HNSW index (same graph, several times faster on a multi-core machine).
parallel = ["rayon", "clawhdf5-ann?/parallel"]
# Compress embeddings with Zstd instead of deflate when
# `MemoryConfig::compression` is on. Off by default: it links libzstd (C).
zstd = ["clawhdf5/zstd"]
@@ -57,7 +68,6 @@ zstd = ["clawhdf5/zstd"]
# `--no-default-features` (plus re-enabling other defaults) to force the exact
# linear cosine scan.
hnsw = ["clawhdf5-ann"]
agent = []
gpu = ["clawhdf5-gpu/gpu-wgpu"]
fast-math = ["matrixmultiply"]
accelerate = ["accelerate-src", "cblas-sys"]
crates/clawhdf5-agent/benches/multimodal_bench.rs
+107
View File
@@ -0,0 +1,107 @@
//! Multi-modal memory search benchmarks (`clawhdf5_agent::multimodal`).
//!
//! Covers `MultiModalStore::search_cross_modal` (every embedding of every
//! record, whatever its modality) and, for comparison,
//! `MultiModalStore::search_by_modality` restricted to one modality.
//!
//! Corpus: N records (1K and 10K), each carrying two 384-dim embeddings —
//! a text embedding of its caption plus one embedding of its primary modality,
//! cycling Image / Audio / Video — so a cross-modal query scores 2N vectors.
//! All data comes from a fixed-seed LCG, so every run sees the same corpus.
//!
//! Run: `cargo bench -p clawhdf5-agent --bench multimodal_bench`
use std::collections::HashMap;
use clawhdf5_agent::multimodal::{
MediaRef, ModalEmbedding, Modality, MultiModalRecord, MultiModalStore,
};
use criterion::{BenchmarkId, Criterion, criterion_group, criterion_main};
// ---------------------------------------------------------------------------
// Simple deterministic PRNG (LCG), same as the other agent benches
// ---------------------------------------------------------------------------
struct Rng(u32);
impl Rng {
fn new(seed: u32) -> Self {
Self(seed)
}
fn next_u32(&mut self) -> u32 {
self.0 = self.0.wrapping_mul(1103515245).wrapping_add(12345);
self.0 >> 16
}
fn next_f32(&mut self) -> f32 {
self.next_u32() as f32 / 65536.0 - 0.5
}
}
fn make_vec(rng: &mut Rng, dim: usize) -> Vec<f32> {
(0..dim).map(|_| rng.next_f32()).collect()
}
// ---------------------------------------------------------------------------
// Corpus
// ---------------------------------------------------------------------------
const DIM: usize = 384;
const K: usize = 10;
const MEDIA: [(Modality, &str, &str); 3] = [
(Modality::Image, "image/png", "clip-vit-base"),
(Modality::Audio, "audio/wav", "clap-base"),
(Modality::Video, "video/mp4", "xclip-base"),
];
fn build_store(n: usize, seed: u32) -> MultiModalStore {
let mut rng = Rng::new(seed);
let mut store = MultiModalStore::new();
for i in 0..n {
let (modality, mime, model) = &MEDIA[i % MEDIA.len()];
let embeddings = vec![
ModalEmbedding::new(Modality::Text, make_vec(&mut rng, DIM), "minilm-l6"),
ModalEmbedding::new(modality.clone(), make_vec(&mut rng, DIM), *model),
];
store.add_record(MultiModalRecord {
id: 0,
primary_modality: modality.clone(),
text_content: Some(format!("{modality} memory {i}")),
media_ref: Some(MediaRef::path(format!("/media/{i}"), *mime)),
embeddings,
observation: None,
timestamp: 1_700_000_000.0 + i as f64,
metadata: HashMap::new(),
});
}
store
}
// ---------------------------------------------------------------------------
// Benchmarks
// ---------------------------------------------------------------------------
fn multimodal_search_benches(c: &mut Criterion) {
let query = make_vec(&mut Rng::new(99), DIM);
let mut group = c.benchmark_group("multimodal_search");
group.sample_size(50);
for (label, n) in [("1k", 1_000usize), ("10k", 10_000)] {
let store = build_store(n, 42);
assert_eq!(store.count(), n);
group.bench_with_input(BenchmarkId::new("cross_modal", label), &n, |b, _| {
b.iter(|| store.search_cross_modal(&query, K));
});
group.bench_with_input(BenchmarkId::new("by_modality_image", label), &n, |b, _| {
b.iter(|| store.search_by_modality(&Modality::Image, &query, K));
});
}
group.finish();
}
criterion_group!(multimodal_benches, multimodal_search_benches);
criterion_main!(multimodal_benches);
crates/clawhdf5-agent/src/agents_md.rs
+4
View File
@@ -118,6 +118,10 @@ mod tests {
created_at: "2025-01-01T00:00:00Z".to_string(),
wal_enabled: false,
wal_max_entries: 500,
quantized_index: false,
hnsw_m: 16,
hnsw_ef_construction: 64,
hnsw_ef_search: 0,
}
}
crates/clawhdf5-agent/src/async_memory.rs
+1 -1
View File
@@ -37,7 +37,7 @@
//! let mem = AsyncHDF5Memory::open_with(path, config).await?;
//! mem.save(entry).await?; // buffered → background writer
//! mem.save_batch(entries).await?; // also buffered
//! let results = mem.hybrid_search(emb, "query".into(), 0.7, 0.3, 5).await;
//! let results = mem.hybrid_search(emb, "query".into(), 0.4, 0.6, 5).await;
//! mem.shutdown().await?; // final flush + stop
//! ```
crates/clawhdf5-agent/src/bm25.rs
+148 -9
View File
@@ -59,11 +59,18 @@ pub struct BM25Index {
k1: f32,
/// BM25 b parameter.
b: f32,
/// Applied to every document and query token, so the two always agree.
filter: TokenFilter,
}
impl BM25Index {
/// Build a BM25 index from a set of documents, excluding tombstoned entries.
pub fn build(documents: &[String], tombstones: &[u8]) -> Self {
Self::build_with(documents, tombstones, TokenFilter::default())
}
/// [`BM25Index::build`] with the token filter chosen explicitly.
pub fn build_with(documents: &[String], tombstones: &[u8], filter: TokenFilter) -> Self {
let mut index = Self {
inverted: HashMap::new(),
doc_lengths: vec![0; documents.len()],
@@ -72,6 +79,7 @@ impl BM25Index {
num_docs: 0,
k1: DEFAULT_K1,
b: DEFAULT_B,
filter,
};
index.index_documents(documents, tombstones);
index
@@ -80,8 +88,11 @@ impl BM25Index {
/// Search the index for a query, returning the top `k` results
/// as `(doc_id, score)` pairs sorted by score descending.
///
/// Uses Block-Max WAND for early termination when remaining documents
/// cannot beat the current top-k threshold.
/// Scores every matching document exhaustively, then keeps the top `k`.
/// There is no early termination (WAND, MaxScore): the store's hot path
/// is [`scores`](Self::scores), because score fusion normalises over the
/// whole matching set and so needs every score, which no pruning scheme
/// can skip. This method is for BM25-only callers.
pub fn search(&self, query: &str, k: usize) -> Vec<(usize, f32)> {
if k == 0 {
return Vec::new();
@@ -120,7 +131,7 @@ impl BM25Index {
// add/remove, and costs one `ln` per query term.
let mut acc = vec![0.0f32; self.doc_lengths.len()];
let mut matched = false;
for token in tokenize(query) {
for token in tokenize_with(query, self.filter) {
let Some(postings) = self.inverted.get(token.as_str()) else {
continue;
};
@@ -146,6 +157,11 @@ impl BM25Index {
.collect()
}
/// The token filter this index was built with.
pub fn token_filter(&self) -> TokenFilter {
self.filter
}
/// Number of document slots (live or not) the index covers. Ids are
/// positions in the document list it mirrors.
pub fn len(&self) -> usize {
@@ -168,7 +184,7 @@ impl BM25Index {
}
debug_assert_eq!(self.doc_lengths[doc_id], 0, "slot {doc_id} is occupied");
let tokens = tokenize(text);
let tokens = tokenize_with(text, self.filter);
let mut term_freqs: HashMap<&str, u32> = HashMap::new();
for token in &tokens {
*term_freqs.entry(token).or_insert(0) += 1;
@@ -201,7 +217,7 @@ impl BM25Index {
/// Remove document `doc_id`, whose indexed text was `text`. The text is
/// needed to find its postings; pass exactly what was added.
pub fn remove_document(&mut self, doc_id: usize, text: &str) {
let tokens = tokenize(text);
let tokens = tokenize_with(text, self.filter);
let mut seen: std::collections::HashSet<&str> = std::collections::HashSet::new();
for token in &tokens {
if !seen.insert(token) {
@@ -252,7 +268,7 @@ impl BM25Index {
continue;
}
let tokens = tokenize(doc);
let tokens = tokenize_with(doc, self.filter);
let doc_len = tokens.len() as u32;
self.doc_lengths[i] = doc_len;
total_length += doc_len as u64;
@@ -285,11 +301,86 @@ impl BM25Index {
/// Tokenize a string: lowercase, split on non-alphanumeric characters,
/// filter empty tokens.
/// What [`tokenize_with`] does to each token after splitting.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
pub enum TokenFilter {
/// Lowercase and split only — the original behaviour.
#[default]
Plain,
/// Also strip common English inflections, so "running" and "runs" match
/// "run". Conservative on purpose: only plural and past/continuous verb
/// endings, and only on tokens long enough that stripping leaves a real
/// stem. A stemmer earns its keep by conflating *related* words; an
/// aggressive one also conflates unrelated ones ("universe"/"university"),
/// which costs precision.
Stemmed,
}
/// Strip common English inflections from an already-lowercased token.
///
/// Applied identically to documents and queries, so the pair only has to agree
/// with itself — the stem need not be a real word.
fn stem(token: &str) -> &str {
// Below this, stripping does more harm than good ("bed" -> "b").
const MIN_STEM: usize = 4;
let strip = |suffix: &str, min_len: usize| -> Option<&str> {
let stem = token.strip_suffix(suffix)?;
(stem.len() >= min_len).then_some(stem)
};
// Plurals first: "studies" -> "studi", "classes" -> "class", "cats" -> "cat".
// "ies" keeps its "i" so the result meets "-ied" ("studied" -> "studi").
if let Some(stem) = strip("ies", 2) {
return &token[..stem.len() + 1];
}
for suffix in ["sses", "shes", "ches", "xes", "zes"] {
if let Some(stem) = strip(suffix, MIN_STEM - 1) {
// Keep the sibilant: "classes" -> "class", not "clas".
return &token[..stem.len() + 2];
}
}
// Verb endings before the bare plural, so "raced" doesn't become "raced".
if let Some(stem) = strip("ing", MIN_STEM - 1).or_else(|| strip("ed", MIN_STEM - 1)) {
return undouble(stem);
}
if !token.ends_with("ss")
&& !token.ends_with("us")
&& !token.ends_with("is")
&& let Some(stem) = strip("s", MIN_STEM - 1)
{
return stem;
}
token
}
/// "runn" -> "run": undo the consonant doubling that "-ing"/"-ed" introduce.
fn undouble(stem: &str) -> &str {
let mut chars = stem.chars().rev();
let (Some(last), Some(prev)) = (chars.next(), chars.next()) else {
return stem;
};
let doubled = last == prev && !"aeiou".contains(last) && last.is_ascii_alphabetic();
if doubled && stem.len() > 3 {
&stem[..stem.len() - 1]
} else {
stem
}
}
#[cfg(test)]
fn tokenize(text: &str) -> Vec<String> {
tokenize_with(text, TokenFilter::Plain)
}
/// Split `text` into scoring tokens under `filter`.
pub fn tokenize_with(text: &str, filter: TokenFilter) -> Vec<String> {
text.to_lowercase()
.split(|c: char| !c.is_alphanumeric())
.filter(|s| !s.is_empty())
.map(|s| s.to_string())
.map(|token| match filter {
TokenFilter::Plain => token.to_string(),
TokenFilter::Stemmed => stem(token).to_string(),
})
.collect()
}
@@ -473,8 +564,9 @@ mod tests {
}
#[test]
fn wand_returns_same_results_as_exhaustive() {
// WAND-style search should produce same scores as exhaustive
fn top_k_search_matches_ranking_every_score() {
// `search` must agree with ranking the full `scores` set — the
// bounded heap is an optimisation over sorting, not an approximation.
let docs: Vec<String> = (0..100)
.map(|i| {
if i % 3 == 0 {
@@ -599,6 +691,53 @@ mod tests {
}
}
#[test]
fn stemming_conflates_inflections_of_the_same_word() {
let stem_of = |w: &str| tokenize_with(w, TokenFilter::Stemmed).pop().unwrap();
// Pairs that should meet.
for (a, b) in [
("running", "runs"),
("trained", "training"),
("miles", "mile"),
("studies", "studied"),
("mentioned", "mentioning"),
("classes", "class"),
("planned", "planning"),
] {
assert_eq!(stem_of(a), stem_of(b), "{a} / {b} should share a stem");
}
// Pairs that must stay apart. Note which pairs are deliberately absent:
// "bed"/"bedding" and "gas"/"gassed" both collapse to one stem, which
// is what Porter does too and is right — they are related words.
for (a, b) in [
("universe", "university"),
("business", "busy"),
("this", "thing"),
] {
assert_ne!(stem_of(a), stem_of(b), "{a} / {b} must not be conflated");
}
// Short words and non-inflections are left alone.
for word in ["run", "bus", "is", "his", "data", "gas"] {
assert_eq!(stem_of(word), word, "{word} should be untouched");
}
}
#[test]
fn stemming_is_off_by_default_and_applied_consistently() {
assert_eq!(tokenize("Running miles"), ["running", "miles"]);
assert_eq!(
tokenize_with("Running miles", TokenFilter::Stemmed),
["run", "mile"]
);
// A query inflected differently from the document still matches.
let docs = vec!["I ran while training for the marathon".to_string()];
let plain = BM25Index::build_with(&docs, &[0], TokenFilter::Plain);
let stemmed = BM25Index::build_with(&docs, &[0], TokenFilter::Stemmed);
assert!(plain.search("trains", 1).is_empty());
assert_eq!(stemmed.search("trains", 1).len(), 1);
}
#[test]
fn ties_break_towards_the_lower_doc_id() {
let docs: Vec<String> = (0..6).map(|_| "same text".to_string()).collect();
crates/clawhdf5-agent/src/cache.rs
+260 -47
View File
@@ -1,17 +1,145 @@
//! In-memory cache for memory entries, sessions, and knowledge graph.
use crate::vector_search;
use clawhdf5_format::float16::round_to_f16;
/// Every entry's embedding, in one contiguous `[N x dim]` buffer.
///
/// Rows are always exactly `dim` long: a shorter one is zero-padded, a longer
/// one truncated. The previous `Vec<Vec<f32>>` allowed ragged rows, which
/// silently misaligned the flattened copy that the batched kernels read — a
/// single wrong-length embedding shifted every row after it. Padding makes
/// that unrepresentable. A record stored without an embedding therefore holds
/// a zero row, and is told apart by its norm being zero rather than by length.
///
/// This used to be two fields — a `Vec<Vec<f32>>` and a flattened copy kept in
/// lock-step — which stored the whole corpus twice and cost one heap
/// allocation per entry on top. At 100k 384-dim entries that duplicate was
/// ~150 MiB. Indexing yields a `&[f32]` row, so `embeddings[i]` still reads
/// the same way.
#[derive(Debug, Clone, Default)]
pub struct Embeddings {
flat: Vec<f32>,
dim: usize,
}
impl Embeddings {
pub fn new(dim: usize) -> Self {
Self {
flat: Vec::new(),
dim,
}
}
/// Number of embeddings.
pub fn len(&self) -> usize {
self.flat.len().checked_div(self.dim).unwrap_or(0)
}
pub fn is_empty(&self) -> bool {
self.len() == 0
}
/// The whole buffer, `[N x dim]` row-major — what batched kernels read.
pub fn as_flat(&self) -> &[f32] {
&self.flat
}
pub fn dim(&self) -> usize {
self.dim
}
/// Row `i`, or `None` if out of range.
pub fn get(&self, i: usize) -> Option<&[f32]> {
let start = i.checked_mul(self.dim)?;
self.flat.get(start..start.checked_add(self.dim)?)
}
pub fn iter(&self) -> impl ExactSizeIterator<Item = &[f32]> {
self.flat.chunks_exact(self.dim.max(1))
}
/// Append one embedding. A row whose length doesn't match `dim` is padded
/// or truncated, so the buffer stays rectangular whatever a caller passes.
pub fn push(&mut self, embedding: &[f32]) {
if self.dim == 0 {
return;
}
let take = embedding.len().min(self.dim);
self.flat.extend_from_slice(&embedding[..take]);
self.flat.resize(self.flat.len() + (self.dim - take), 0.0);
}
/// Replace row `i`. Out-of-range indices are ignored.
pub fn set(&mut self, i: usize, embedding: &[f32]) {
let Some(start) = i.checked_mul(self.dim) else {
return;
};
if start + self.dim > self.flat.len() {
return;
}
let take = embedding.len().min(self.dim);
self.flat[start..start + take].copy_from_slice(&embedding[..take]);
self.flat[start + take..start + self.dim].fill(0.0);
}
/// Keep only the rows `keep` returns true for, preserving order.
pub fn retain(&mut self, mut keep: impl FnMut(usize) -> bool) {
if self.dim == 0 {
return;
}
let mut write = 0usize;
for read in 0..self.len() {
if keep(read) {
if write != read {
let (dst, src) = (write * self.dim, read * self.dim);
self.flat.copy_within(src..src + self.dim, dst);
}
write += 1;
}
}
self.flat.truncate(write * self.dim);
}
/// Replace the contents with `rows`.
pub fn reset_from(&mut self, dim: usize, rows: impl IntoIterator<Item = Vec<f32>>) {
self.dim = dim;
self.flat.clear();
for row in rows {
self.push(&row);
}
}
/// Adopt an already-flat buffer, trimming any partial trailing row.
pub fn set_flat(&mut self, dim: usize, mut flat: Vec<f32>) {
self.dim = dim;
match flat.len().checked_div(dim) {
Some(rows) => flat.truncate(rows * dim),
None => flat.clear(),
}
self.flat = flat;
}
}
impl PartialEq for Embeddings {
fn eq(&self, other: &Self) -> bool {
self.dim == other.dim && self.flat == other.flat
}
}
impl std::ops::Index<usize> for Embeddings {
type Output = [f32];
fn index(&self, i: usize) -> &[f32] {
self.get(i).expect("embedding index out of range")
}
}
/// In-memory cache for the /memory group data.
#[derive(Debug, Clone)]
pub struct MemoryCache {
pub chunks: Vec<String>,
pub embeddings: Vec<Vec<f32>>,
/// `embeddings` flattened into one contiguous `[N × embedding_dim]`
/// buffer, maintained incrementally alongside `embeddings` (push/update/
/// compact) so BLAS/Accelerate batch search can read it directly instead
/// of re-flattening the whole corpus on every query.
pub embeddings_flat: Vec<f32>,
pub embeddings: Embeddings,
pub source_channels: Vec<String>,
pub timestamps: Vec<f64>,
pub session_ids: Vec<String>,
@@ -22,14 +150,18 @@ pub struct MemoryCache {
pub norms: Vec<f32>,
/// Hebbian activation weights (default 1.0 per entry).
pub activation_weights: Vec<f32>,
/// Round every embedding to IEEE half precision as it enters the cache,
/// so the cache holds exactly what a `float16` store writes to disk. Set
/// it with [`MemoryCache::set_half_precision`], which also rounds the
/// rows already held.
pub half_precision: bool,
}
impl MemoryCache {
pub fn new(embedding_dim: usize) -> Self {
Self {
chunks: Vec::new(),
embeddings: Vec::new(),
embeddings_flat: Vec::new(),
embeddings: Embeddings::new(embedding_dim),
source_channels: Vec::new(),
timestamps: Vec::new(),
session_ids: Vec::new(),
@@ -38,18 +170,52 @@ impl MemoryCache {
embedding_dim,
norms: Vec::new(),
activation_weights: Vec::new(),
half_precision: false,
}
}
/// Rebuild `embeddings_flat` from `embeddings` from scratch. Callers that
/// populate `embeddings` directly (bulk loads) must call this afterward.
pub fn rebuild_flat(&mut self) {
self.embeddings_flat.clear();
self.embeddings_flat
.reserve(self.embeddings.len() * self.embedding_dim);
for emb in &self.embeddings {
self.embeddings_flat.extend_from_slice(emb);
/// Switch half-precision rounding on or off. Turning it on rounds every
/// embedding already held (and recomputes norms where one changed) —
/// e.g. a `float16` store whose last checkpoint predates half-precision
/// storage and so is still `f32` on disk.
pub fn set_half_precision(&mut self, on: bool) {
self.half_precision = on;
if !on {
return;
}
for i in 0..self.embeddings.len() {
let row = &self.embeddings[i];
if row
.iter()
.all(|&v| round_to_f16(v).to_bits() == v.to_bits())
{
continue;
}
let rounded: Vec<f32> = row.iter().map(|&v| round_to_f16(v)).collect();
self.norms[i] = vector_search::compute_norm(&rounded);
self.embeddings.set(i, &rounded);
}
}
/// The embedding as the cache will hold it: rounded to half precision
/// when [`Self::half_precision`] is on, otherwise unchanged.
fn stored_form(&self, mut embedding: Vec<f32>) -> Vec<f32> {
if self.half_precision {
for v in &mut embedding {
*v = round_to_f16(*v);
}
}
embedding
}
/// Kept for callers that used to have to re-flatten after a bulk load.
/// The buffer is always flat now, so there is nothing to rebuild.
#[deprecated(note = "embeddings are stored flat; this is a no-op")]
pub fn rebuild_flat(&mut self) {}
/// The embeddings as one contiguous `[N x dim]` buffer.
pub fn flat_embeddings(&self) -> &[f32] {
self.embeddings.as_flat()
}
/// Total number of entries (including tombstoned).
@@ -77,10 +243,10 @@ impl MemoryCache {
tags: String,
) -> usize {
let idx = self.chunks.len();
let embedding = self.stored_form(embedding);
let norm = vector_search::compute_norm(&embedding);
self.chunks.push(chunk);
self.embeddings_flat.extend_from_slice(&embedding);
self.embeddings.push(embedding);
self.embeddings.push(&embedding);
self.source_channels.push(source_channel);
self.timestamps.push(timestamp);
self.session_ids.push(session_id);
@@ -116,22 +282,10 @@ impl MemoryCache {
session_id: String,
) {
if idx < self.chunks.len() {
let embedding = self.stored_form(embedding);
let norm = vector_search::compute_norm(&embedding);
self.chunks[idx] = chunk;
let dim = self.embedding_dim;
let flat_start = idx * dim;
let matches_dim =
embedding.len() == dim && flat_start + dim <= self.embeddings_flat.len();
self.embeddings[idx] = embedding;
if matches_dim {
self.embeddings_flat[flat_start..flat_start + dim]
.copy_from_slice(&self.embeddings[idx]);
} else {
// Embedding length doesn't match embedding_dim (shouldn't
// happen in practice) — fall back to a full rebuild rather
// than leave embeddings_flat misaligned with embeddings.
self.rebuild_flat();
}
self.embeddings.set(idx, &embedding);
self.source_channels[idx] = source_channel;
self.timestamps[idx] = timestamp;
self.session_ids[idx] = session_id;
@@ -183,7 +337,7 @@ impl MemoryCache {
new_idx += 1;
let norm = vector_search::compute_norm(&self.embeddings[i]);
new_chunks.push(self.chunks[i].clone());
new_embeddings.push(self.embeddings[i].clone());
new_embeddings.push(self.embeddings[i].to_vec());
new_source_channels.push(self.source_channels[i].clone());
new_timestamps.push(self.timestamps[i]);
new_session_ids.push(self.session_ids[i].clone());
@@ -196,7 +350,8 @@ impl MemoryCache {
let removed = old_len - new_chunks.len();
self.chunks = new_chunks;
self.embeddings = new_embeddings;
self.embeddings
.reset_from(self.embedding_dim, new_embeddings);
self.source_channels = new_source_channels;
self.timestamps = new_timestamps;
self.session_ids = new_session_ids;
@@ -204,16 +359,14 @@ impl MemoryCache {
self.tombstones = new_tombstones;
self.norms = new_norms;
self.activation_weights = new_activation_weights;
self.rebuild_flat();
(removed, index_map)
}
/// Flatten all embeddings into a single Vec<f32> for HDF5 storage.
/// `embeddings_flat` is already maintained incrementally, so this just
/// clones it — kept as a method for callers that want an owned copy.
pub fn flat_embeddings(&self) -> Vec<f32> {
self.embeddings_flat.clone()
/// All embeddings as one owned `[N x dim]` buffer, for HDF5 storage.
/// Prefer [`MemoryCache::flat_embeddings`] where a borrow will do.
pub fn flat_embeddings_owned(&self) -> Vec<f32> {
self.embeddings.as_flat().to_vec()
}
}
@@ -224,7 +377,7 @@ mod tests {
/// `embeddings_flat` must always equal a from-scratch flatten of `embeddings`.
fn assert_flat_in_sync(cache: &MemoryCache) {
let expected: Vec<f32> = cache.embeddings.iter().flatten().copied().collect();
assert_eq!(cache.embeddings_flat, expected);
assert_eq!(cache.embeddings.as_flat(), expected);
}
#[test]
@@ -247,7 +400,10 @@ mod tests {
String::new(),
);
assert_flat_in_sync(&cache);
assert_eq!(cache.embeddings_flat, vec![1.0, 2.0, 3.0, 4.0, 5.0, 6.0]);
assert_eq!(
cache.embeddings.as_flat(),
vec![1.0, 2.0, 3.0, 4.0, 5.0, 6.0]
);
}
#[test]
@@ -279,7 +435,7 @@ mod tests {
);
assert_flat_in_sync(&cache);
assert_eq!(
cache.embeddings_flat,
cache.embeddings.as_flat(),
vec![7.0, 8.0, 9.0, 4.0, 5.0, 6.0],
"update must overwrite the correct flat slice, not just append"
);
@@ -315,14 +471,71 @@ mod tests {
cache.mark_deleted(1);
cache.compact();
assert_flat_in_sync(&cache);
assert_eq!(cache.embeddings_flat, vec![1.0, 1.0, 3.0, 3.0]);
assert_eq!(cache.embeddings.as_flat(), vec![1.0, 1.0, 3.0, 3.0]);
}
#[test]
fn rebuild_flat_matches_manual_flatten() {
let mut cache = MemoryCache::new(2);
cache.embeddings = vec![vec![1.0, 2.0], vec![3.0, 4.0]];
cache.rebuild_flat();
assert_eq!(cache.embeddings_flat, vec![1.0, 2.0, 3.0, 4.0]);
cache
.embeddings
.reset_from(2, vec![vec![1.0, 2.0], vec![3.0, 4.0]]);
assert_eq!(cache.embeddings.as_flat(), vec![1.0, 2.0, 3.0, 4.0]);
}
#[test]
fn set_half_precision_rounds_existing_rows_and_their_norms() {
// A store with float16 set whose checkpoint is still f32 on disk
// loads full-precision rows; switching rounding on must bring them to
// exactly what the next checkpoint will write.
let mut cache = MemoryCache::new(3);
cache.push(
"a".into(),
vec![0.1, 0.2, 0.3],
"c".into(),
0.0,
"s".into(),
"".into(),
);
cache.push(
"b".into(),
vec![0.5, 0.25, 1.0],
"c".into(),
0.0,
"s".into(),
"".into(),
);
let exact_norm = cache.norms[0];
cache.set_half_precision(true);
let row0: Vec<f32> = [0.1f32, 0.2, 0.3]
.iter()
.map(|&v| round_to_f16(v))
.collect();
assert_eq!(&cache.embeddings[0], row0.as_slice());
assert_eq!(cache.norms[0], vector_search::compute_norm(&row0));
assert_ne!(cache.norms[0], exact_norm);
// Already representable: untouched.
assert_eq!(&cache.embeddings[1], &[0.5, 0.25, 1.0]);
// New rows are rounded as they arrive, and updates too.
cache.push(
"c".into(),
vec![0.1, 0.0, 0.0],
"c".into(),
0.0,
"s".into(),
"".into(),
);
assert_eq!(cache.embeddings[2][0], round_to_f16(0.1));
cache.update(
2,
"c".into(),
vec![0.3, 0.0, 0.0],
"c".into(),
0.0,
"s".into(),
);
assert_eq!(cache.embeddings[2][0], round_to_f16(0.3));
}
}
crates/clawhdf5-agent/src/consolidation.rs
+126 -2
View File
@@ -144,9 +144,44 @@ pub struct ConsolidationStats {
pub struct ImportanceScorer;
/// Sum of squares, in 8-wide lanes so it vectorises.
fn sum_of_squares(a: &[f32]) -> f32 {
let (blocks, tail) = a.as_chunks::<8>();
let mut acc = [0.0f32; 8];
for b in blocks {
for i in 0..8 {
acc[i] += b[i] * b[i];
}
}
acc.iter().sum::<f32>() + tail.iter().map(|x| x * x).sum::<f32>()
}
/// `(a · b, |b|²)` in one pass over equal-length slices, in 8-wide lanes.
fn dot_and_norm2(a: &[f32], b: &[f32]) -> (f32, f32) {
let (a_blocks, a_tail) = a.as_chunks::<8>();
let (b_blocks, b_tail) = b.as_chunks::<8>();
let mut dot = [0.0f32; 8];
let mut nb = [0.0f32; 8];
for (x, y) in a_blocks.iter().zip(b_blocks) {
for i in 0..8 {
dot[i] += x[i] * y[i];
nb[i] += y[i] * y[i];
}
}
let mut d = dot.iter().sum::<f32>();
let mut n = nb.iter().sum::<f32>();
for (x, y) in a_tail.iter().zip(b_tail) {
d += x * y;
n += y * y;
}
(d, n)
}
impl ImportanceScorer {
/// Cosine similarity between two embedding slices.
/// Returns 0.0 if either norm is zero.
/// Returns 0.0 if either norm is zero. The reference that
/// [`Self::score_surprise`] is tested against.
#[cfg(test)]
fn cosine_similarity(a: &[f32], b: &[f32]) -> f32 {
let len = a.len().min(b.len());
if len == 0 {
@@ -167,13 +202,54 @@ impl ImportanceScorer {
/// Novelty score: 1.0 − max cosine similarity against all existing records.
/// Returns 1.0 when there are no existing memories.
///
/// Same result as the reference cosine similarity against each record, but the
/// new embedding's norm is computed once rather than per record, each
/// record costs one fused pass (dot product and its norm together) rather
/// than three, and a large working set is scored in parallel. Every insert
/// scores against the whole working tier, so this is what an unbounded
/// working tier pays for: at 100K records it was the difference between a
/// benchmark finishing and not (`BENCHMARKS.md`, "Consolidation Efficiency").
pub fn score_surprise(embedding: &[f32], existing_memories: &[&MemoryRecord]) -> f32 {
if existing_memories.is_empty() {
return 1.0;
}
let query_norm2 = sum_of_squares(embedding);
let similarity = |r: &&MemoryRecord| -> f32 {
let other = &r.embedding;
let len = embedding.len().min(other.len());
if len == 0 {
return 0.0;
}
let (dot, other_norm2) = dot_and_norm2(&embedding[..len], &other[..len]);
// A shorter record compares against the query's matching prefix.
let q2 = if len == embedding.len() {
query_norm2
} else {
sum_of_squares(&embedding[..len])
};
if q2 == 0.0 || other_norm2 == 0.0 {
return 0.0;
}
dot / (q2.sqrt() * other_norm2.sqrt())
};
#[cfg(feature = "parallel")]
let max_sim = if existing_memories.len() >= 4096 {
use rayon::prelude::*;
existing_memories
.par_iter()
.map(similarity)
.reduce(|| f32::NEG_INFINITY, f32::max)
} else {
existing_memories
.iter()
.map(similarity)
.fold(f32::NEG_INFINITY, f32::max)
};
#[cfg(not(feature = "parallel"))]
let max_sim = existing_memories
.iter()
.map(|r| Self::cosine_similarity(embedding, &r.embedding))
.map(similarity)
.fold(f32::NEG_INFINITY, f32::max);
(1.0 - max_sim).clamp(0.0, 1.0)
}
@@ -471,6 +547,54 @@ impl ConsolidationEngine {
mod tests {
use super::*;
#[test]
fn score_surprise_matches_the_reference_cosine() {
let mut x = 0x2545_F491_4F6C_DD1Du64;
let mut next = || {
x ^= x << 13;
x ^= x >> 7;
x ^= x << 17;
(x >> 40) as f32 / (1u64 << 24) as f32 - 0.5
};
let make = |id: u64, v: Vec<f32>| MemoryRecord {
id,
chunk: String::new(),
embedding: v,
tier: MemoryTier::Working,
importance: 0.0,
access_count: 0,
last_accessed: 0.0,
created_at: 0.0,
source: MemorySource::User,
};
// Ordinary rows, a shorter one, an empty one and a zero vector; and
// enough rows to take the parallel path too.
for n in [5usize, 5000] {
let mut recs: Vec<MemoryRecord> = (0..n as u64)
.map(|i| make(i, (0..37).map(|_| next()).collect()))
.collect();
recs.push(make(9_000, (0..20).map(|_| next()).collect()));
recs.push(make(9_001, Vec::new()));
recs.push(make(9_002, vec![0.0; 37]));
let refs: Vec<&MemoryRecord> = recs.iter().collect();
for _ in 0..5 {
let q: Vec<f32> = (0..37).map(|_| next()).collect();
let expected = (1.0
- refs
.iter()
.map(|r| ImportanceScorer::cosine_similarity(&q, &r.embedding))
.fold(f32::NEG_INFINITY, f32::max))
.clamp(0.0, 1.0);
let got = ImportanceScorer::score_surprise(&q, &refs);
assert!((got - expected).abs() < 1e-5, "n={n}: {got} vs {expected}");
}
}
assert_eq!(
ImportanceScorer::score_surprise(&[0.0; 4], &[&make(1, vec![1.0; 4])]),
1.0
);
}
// Helper: build a simple normalised embedding of given dimension.
fn unit_vec(dim: usize, hot: usize) -> Vec<f32> {
let mut v = vec![0.0f32; dim];
crates/clawhdf5-agent/src/hybrid.rs
+183 -36
View File
@@ -28,39 +28,69 @@ use crate::vector_search;
pub fn hybrid_search(
query_embedding: &[f32],
query_text: &str,
vectors: &[Vec<f32>],
_chunks: &[String],
vectors: &(impl crate::vector_search::VectorSet + Sync + ?Sized),
chunks: &[String],
tombstones: &[u8],
bm25_index: &BM25Index,
vector_weight: f32,
keyword_weight: f32,
k: usize,
) -> Vec<(usize, f32)> {
hybrid_search_fused(
query_embedding,
query_text,
vectors,
chunks,
tombstones,
bm25_index,
Fusion::Weighted {
vector: vector_weight,
keyword: keyword_weight,
},
k,
)
}
/// [`hybrid_search`] with the fusion method chosen explicitly.
#[allow(clippy::too_many_arguments)]
pub fn hybrid_search_fused(
query_embedding: &[f32],
query_text: &str,
vectors: &(impl crate::vector_search::VectorSet + Sync + ?Sized),
_chunks: &[String],
tombstones: &[u8],
bm25_index: &BM25Index,
fusion: Fusion,
k: usize,
) -> Vec<(usize, f32)> {
// Get raw scores from both systems. Request all results so normalization
// covers the full distribution.
// Use parallel search when rayon feature is enabled and vector count > 10K.
let vec_scores = {
#[cfg(feature = "parallel")]
{
if vectors.len() > 10_000 {
vector_search::parallel_cosine_batch(
query_embedding,
vectors,
tombstones,
vectors.len(),
)
} else {
vector_search::cosine_similarity_batch(query_embedding, vectors, tombstones)
}
}
#[cfg(not(feature = "parallel"))]
{
vector_search::cosine_similarity_batch(query_embedding, vectors, tombstones)
}
};
let vec_scores = exact_vector_scores(query_embedding, vectors, tombstones);
let kw_scores = bm25_index.scores(query_text);
merge_vector_keyword(vec_scores, kw_scores, vector_weight, keyword_weight, k)
fuse(vec_scores, kw_scores, fusion, k)
}
/// Cosine similarity of `query_embedding` to every vector whose `skip` byte is
/// 0 (a tombstone, or any other exclusion mask). Parallel above 10K vectors
/// when the `parallel` feature is on.
pub fn exact_vector_scores(
query_embedding: &[f32],
vectors: &(impl crate::vector_search::VectorSet + Sync + ?Sized),
skip: &[u8],
) -> Vec<(usize, f32)> {
#[cfg(feature = "parallel")]
{
if vectors.count() > 10_000 {
return vector_search::parallel_cosine_batch(
query_embedding,
vectors,
skip,
vectors.count(),
);
}
}
vector_search::cosine_similarity_batch(query_embedding, vectors, skip)
}
/// Merge pre-computed vector-similarity and keyword scores into a single ranking.
@@ -76,18 +106,92 @@ pub fn merge_vector_keyword(
keyword_weight: f32,
k: usize,
) -> Vec<(usize, f32)> {
// Normalize each set to [0, 1].
let vec_normalized = normalize_scores(&vec_scores);
let kw_normalized = normalize_scores(&kw_scores);
fuse(
vec_scores,
kw_scores,
Fusion::Weighted {
vector: vector_weight,
keyword: keyword_weight,
},
k,
)
}
/// How the vector and keyword stages are combined into one ranking.
#[derive(Debug, Clone, Copy, PartialEq)]
pub enum Fusion {
/// Min-max normalise each stage over its own candidates, then take a
/// weighted sum. Uses the *scores*, so a stage that separates its
/// candidates sharply keeps that separation — and a stage whose candidates
/// are all near-identical contributes little.
Weighted {
/// Weight on the vector stage.
vector: f32,
/// Weight on the keyword stage.
keyword: f32,
},
/// Reciprocal rank fusion: each stage contributes `1 / (k + rank)`,
/// ignoring score magnitudes entirely. Robust when the two stages'
/// scores aren't comparable, at the cost of discarding confidence.
Rrf {
/// The rank-damping constant; 60 is the value from the original paper.
k: f32,
},
}
impl Default for Fusion {
fn default() -> Self {
DEFAULT_FUSION
}
}
/// The fusion `hybrid_search` uses unless told otherwise.
///
/// The weights are not a guess: a sweep of every 0.1 step over the full
/// LongMemEval haystack (500 questions, real MiniLM embeddings) found the
/// long-standing 0.7/0.3 default *strictly dominated* — 0.4/0.6 is better at
/// Hit@1, Hit@5, Hit@10 and MRR, at both turn and session granularity. See
/// `BENCHMARKS.md`, "Weight sweep".
pub const DEFAULT_FUSION: Fusion = Fusion::Weighted {
vector: 0.4,
keyword: 0.6,
};
// Merge scores with weights.
/// Combine one ranked candidate list from each stage into a single top-`k`.
///
/// Neither list need be sorted; both are consumed.
pub fn fuse(
vec_scores: Vec<(usize, f32)>,
kw_scores: Vec<(usize, f32)>,
fusion: Fusion,
k: usize,
) -> Vec<(usize, f32)> {
let mut merged: HashMap<usize, f32> = HashMap::new();
for (idx, score) in &vec_normalized {
*merged.entry(*idx).or_insert(0.0) += vector_weight * score;
match fusion {
Fusion::Weighted { vector, keyword } => {
// Normalize each set to [0, 1].
for (idx, score) in &normalize_scores(&vec_scores) {
*merged.entry(*idx).or_insert(0.0) += vector * score;
}
for (idx, score) in &normalize_scores(&kw_scores) {
*merged.entry(*idx).or_insert(0.0) += keyword * score;
}
}
Fusion::Rrf { k: damping } => {
for mut stage in [vec_scores, kw_scores] {
// Rank 1 is the best score. Ties break by index so a stage's
// contribution doesn't depend on the candidate order it
// happened to be produced in.
stage.sort_by(|a, b| {
b.1.partial_cmp(&a.1)
.unwrap_or(std::cmp::Ordering::Equal)
.then(a.0.cmp(&b.0))
});
for (rank, (idx, _)) in stage.iter().enumerate() {
*merged.entry(*idx).or_insert(0.0) += 1.0 / (damping + (rank + 1) as f32);
}
}
}
for (idx, score) in &kw_normalized {
*merged.entry(*idx).or_insert(0.0) += keyword_weight * score;
}
let mut results: Vec<(usize, f32)> = merged.into_iter().collect();
@@ -169,7 +273,7 @@ fn normalize_scores(scores: &[(usize, f32)]) -> Vec<(usize, f32)> {
pub fn rrf_hybrid_search(
query_embedding: &[f32],
query_text: &str,
vectors: &[Vec<f32>],
vectors: &(impl crate::vector_search::VectorSet + Sync + ?Sized),
_chunks: &[String],
tombstones: &[u8],
bm25_index: &BM25Index,
@@ -181,12 +285,12 @@ pub fn rrf_hybrid_search(
let mut vec_scores = {
#[cfg(feature = "parallel")]
{
if vectors.len() > 10_000 {
if vectors.count() > 10_000 {
vector_search::parallel_cosine_batch(
query_embedding,
vectors,
tombstones,
vectors.len(),
vectors.count(),
)
} else {
vector_search::cosine_similarity_batch(query_embedding, vectors, tombstones)
@@ -197,7 +301,7 @@ pub fn rrf_hybrid_search(
vector_search::cosine_similarity_batch(query_embedding, vectors, tombstones)
}
};
let mut kw_scores = bm25_index.search(query_text, vectors.len());
let mut kw_scores = bm25_index.search(query_text, vectors.count());
// Sort both lists descending so rank 1 = best.
vec_scores.sort_by(|a, b| b.1.partial_cmp(&a.1).unwrap_or(std::cmp::Ordering::Equal));
@@ -353,6 +457,49 @@ mod tests {
assert_eq!(result[0].1, 1.0);
}
#[test]
fn default_fusion_is_the_tuned_operating_point() {
// A sweep over the full LongMemEval haystack found 0.7/0.3 strictly
// dominated by 0.4/0.6 (BENCHMARKS.md). This guards the finding
// against being quietly undone.
assert_eq!(
DEFAULT_FUSION,
Fusion::Weighted {
vector: 0.4,
keyword: 0.6
}
);
}
#[test]
fn rrf_rewards_agreement_between_the_stages_and_ignores_magnitudes() {
// Doc 1 is second-best in both stages; doc 0 is best in one and absent
// from the other. RRF prefers the doc both stages liked.
let vec_scores = vec![(0, 100.0), (1, 0.9)];
let kw_scores = vec![(2, 5.0), (1, 4.9)];
let ranked = fuse(vec_scores, kw_scores, Fusion::Rrf { k: 60.0 }, 3);
assert_eq!(ranked[0].0, 1, "{ranked:?}");
// Scaling one stage's scores cannot change an RRF ranking, only the
// order within that stage can.
let a = fuse(
vec![(0, 1.0), (1, 0.5)],
vec![(1, 2.0), (0, 1.0)],
Fusion::Rrf { k: 60.0 },
2,
);
let b = fuse(
vec![(0, 1e6), (1, -3.0)],
vec![(1, 0.002), (0, 0.001)],
Fusion::Rrf { k: 60.0 },
2,
);
assert_eq!(
a.iter().map(|r| r.0).collect::<Vec<_>>(),
b.iter().map(|r| r.0).collect::<Vec<_>>()
);
}
#[test]
fn merge_top_k_matches_a_full_sort() {
// Many ties (scores repeat) so the index tie-break is exercised.
crates/clawhdf5-agent/src/knowledge.rs
+115 -8
View File
@@ -163,12 +163,13 @@ fn levenshtein(a: &str, b: &str) -> usize {
/// entities-slice-index map, and an entity-id -> relation-indices map (edges
/// touching that entity as either source or target).
///
/// Built fresh per traversal call rather than cached on `KnowledgeCache`:
/// entities/relations are plain `pub` `Vec`s that get pushed to directly
/// (e.g. `schema.rs`'s load path bypasses `add_entity`/`add_relation`), so a
/// persistent index would need extra bookkeeping to avoid drifting stale. A
/// one-off O(V+E) build per call is still a large win over the O(V·E) (BFS)
/// / O(steps·active·E) (spreading activation) scans it replaces.
/// Cached on `KnowledgeCache` and checked against a fingerprint of the graph
/// on every use ([`graph_fingerprint`]). entities/relations are plain `pub`
/// `Vec`s that get changed directly (e.g. `schema.rs`'s load path bypasses
/// `add_entity`/`add_relation`), so the cache cannot rely on being told about
/// changes; the fingerprint notices any of them. Rebuilding it on every
/// traversal instead made a 2-hop BFS over 1K entities 6.5x slower than the
/// scan it replaced (24 -> 155 µs; `BENCHMARKS.md`, "Knowledge Graph").
struct AdjacencyIndex {
entity_index: HashMap<u64, usize>,
by_entity: HashMap<u64, Vec<usize>>,
@@ -204,6 +205,45 @@ impl AdjacencyIndex {
}
}
/// A hash of everything [`AdjacencyIndex`] depends on — each entity's id and
/// position, each relation's endpoints and position. One linear pass, no
/// allocation: far cheaper than building the index, which hashes the same
/// values into two maps.
fn graph_fingerprint(entities: &[Entity], relations: &[Relation]) -> u64 {
// splitmix64-style mixing; order matters, so positions are covered.
fn mix(h: u64, v: u64) -> u64 {
let mut z = (h ^ v).wrapping_add(0x9E37_79B9_7F4A_7C15);
z = (z ^ (z >> 30)).wrapping_mul(0xBF58_476D_1CE4_E5B9);
z = (z ^ (z >> 27)).wrapping_mul(0x94D0_49BB_1331_11EB);
z ^ (z >> 31)
}
let mut h = mix(entities.len() as u64, relations.len() as u64);
for e in entities {
h = mix(h, e.id);
}
for r in relations {
h = mix(mix(h, r.src), r.tgt);
}
h
}
/// The cached [`AdjacencyIndex`] and the fingerprint it was built for.
/// Cloning a `KnowledgeCache` starts the clone with an empty cache.
#[derive(Default)]
struct AdjacencyCache(std::sync::Mutex<Option<(u64, std::sync::Arc<AdjacencyIndex>)>>);
impl Clone for AdjacencyCache {
fn clone(&self) -> Self {
Self::default()
}
}
impl std::fmt::Debug for AdjacencyCache {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.write_str("AdjacencyCache")
}
}
// ---------------------------------------------------------------------------
// KnowledgeCache
// ---------------------------------------------------------------------------
@@ -216,6 +256,7 @@ pub struct KnowledgeCache {
pub alias_strings: Vec<String>,
pub alias_entity_ids: Vec<i64>,
next_entity_id: u64,
adjacency: AdjacencyCache,
}
impl KnowledgeCache {
@@ -226,6 +267,7 @@ impl KnowledgeCache {
alias_strings: Vec::new(),
alias_entity_ids: Vec::new(),
next_entity_id: 0,
adjacency: AdjacencyCache::default(),
}
}
@@ -236,9 +278,29 @@ impl KnowledgeCache {
alias_strings: Vec::new(),
alias_entity_ids: Vec::new(),
next_entity_id: next_id,
adjacency: AdjacencyCache::default(),
}
}
/// The adjacency index for the graph as it is now: the cached one if the
/// graph's fingerprint still matches, otherwise rebuilt and cached.
fn adjacency_index(&self) -> std::sync::Arc<AdjacencyIndex> {
let fp = graph_fingerprint(&self.entities, &self.relations);
let mut slot = self
.adjacency
.0
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner);
if let Some((cached_fp, idx)) = slot.as_ref()
&& *cached_fp == fp
{
return idx.clone();
}
let idx = std::sync::Arc::new(AdjacencyIndex::build(&self.entities, &self.relations));
*slot = Some((fp, idx.clone()));
idx
}
// -----------------------------------------------------------------------
// Entity management
// -----------------------------------------------------------------------
@@ -397,7 +459,7 @@ impl KnowledgeCache {
/// together with their discovered depth. The seed entity itself is NOT
/// included. Traversal follows both outgoing and incoming relation edges.
pub fn bfs_neighbors(&self, entity_id: u64, max_depth: usize) -> Vec<(Entity, usize)> {
let idx = AdjacencyIndex::build(&self.entities, &self.relations);
let idx = self.adjacency_index();
let mut visited: HashSet<u64> = HashSet::new();
let mut queue: VecDeque<(u64, usize)> = VecDeque::new();
let mut results: Vec<(Entity, usize)> = Vec::new();
@@ -502,7 +564,7 @@ impl KnowledgeCache {
min_activation: f32,
max_steps: usize,
) -> Vec<(u64, f32)> {
let idx = AdjacencyIndex::build(&self.entities, &self.relations);
let idx = self.adjacency_index();
let mut activation: HashMap<u64, f32> = HashMap::new();
// Initialise seeds with activation 1.0.
@@ -631,6 +693,51 @@ impl Default for KnowledgeCache {
mod tests {
use super::*;
#[test]
fn cached_adjacency_sees_direct_changes_to_the_graph() {
// The index is cached across traversals, but entities/relations are
// pub Vecs anyone can edit; every kind of edit must be seen.
let mut kg = KnowledgeCache::new();
let a = kg.add_entity("a", "t", -1);
let b = kg.add_entity("b", "t", -1);
let c = kg.add_entity("c", "t", -1);
kg.add_relation(a, b, "r", 1.0);
let ids = |kg: &KnowledgeCache| -> Vec<u64> {
let mut v: Vec<u64> = kg.bfs_neighbors(a, 3).iter().map(|(e, _)| e.id).collect();
v.sort();
v
};
assert_eq!(ids(&kg), vec![b]);
assert_eq!(ids(&kg), vec![b], "cached index reused");
// Pushed directly, bypassing add_relation.
kg.relations.push(Relation {
src: b,
tgt: c,
..Relation::default()
});
assert_eq!(ids(&kg), vec![b, c]);
// Rewired in place: same lengths, different edge.
kg.relations[1].tgt = a;
assert_eq!(ids(&kg), vec![b]);
// Removed and replaced: same lengths again.
kg.relations.pop();
kg.relations.push(Relation {
src: a,
tgt: c,
..Relation::default()
});
assert_eq!(ids(&kg), vec![b, c]);
let act: Vec<u64> = kg
.spreading_activation(&[a], 0.5, 0.0, 2)
.iter()
.map(|(id, _)| *id)
.collect();
assert!(act.contains(&c));
}
// -----------------------------------------------------------------------
// Original tests — must remain passing
// -----------------------------------------------------------------------
crates/clawhdf5-agent/src/lib.rs
+404 -25
View File
@@ -1,4 +1,4 @@
//! ZeroClaw agent memory HDF5 backend.
//! Agent memory stored in a single HDF5 file.
//!
//! Provides persistent memory storage for AI agents using HDF5 files.
//! All data is cached in-memory for fast access and flushed to disk
@@ -36,6 +36,7 @@ pub mod reranker;
pub mod schema;
pub mod search;
pub mod session;
pub mod signing;
pub mod storage;
mod store_lock;
pub mod temporal;
@@ -62,26 +63,23 @@ use std::path::{Path, PathBuf};
use cache::MemoryCache;
#[cfg(feature = "hnsw")]
use clawhdf5_ann::{DistanceMetric, HnswIndex};
use clawhdf5_ann::{DistanceMetric, HnswIndex, Storage};
use clawhdf5_format::float16::round_to_f16;
use ephemeral::{EphemeralConfig, EphemeralStore};
/// HNSW construction parameters used for the agent's vector index. Cosine is the
/// agent's similarity metric, so the index is built with cosine distance.
#[cfg(feature = "hnsw")]
const HNSW_M: usize = 16;
#[cfg(feature = "hnsw")]
const HNSW_EF_CONSTRUCTION: usize = 64;
// EphemeralEntry and EphemeralStats are part of the crate public API via
// the `ephemeral` module; they are not needed directly in lib.rs internals.
#[allow(unused_imports)]
pub use ephemeral::{EphemeralEntry, EphemeralStats};
use knowledge::KnowledgeCache;
use memory_strategy::{Exchange, MemoryStrategy, StrategyOutput};
use session::SessionCache;
pub use search::SearchOptions;
pub use session::{SessionCache, SessionEntry};
// --- Error type ---
#[derive(Debug)]
#[non_exhaustive]
pub enum MemoryError {
Io(std::io::Error),
Hdf5(String),
@@ -89,6 +87,14 @@ pub enum MemoryError {
NotFound(String),
/// Another `HDF5Memory` (in this or another process) has the store open.
Locked(String),
/// A record the store cannot hold as given, e.g. an embedding value
/// outside the half-precision range of a `float16` store.
InvalidEntry(String),
/// The store's checkpoints are signed and no signing key is set, so a
/// checkpoint would leave it unsigned. Set the key with
/// [`HDF5Memory::set_signing_key`], or drop the signature on purpose with
/// [`HDF5Memory::remove_signature`].
SigningKeyRequired(String),
}
impl std::fmt::Display for MemoryError {
@@ -99,6 +105,8 @@ impl std::fmt::Display for MemoryError {
MemoryError::Schema(e) => write!(f, "schema error: {e}"),
MemoryError::NotFound(e) => write!(f, "not found: {e}"),
MemoryError::Locked(e) => write!(f, "store is locked: {e}"),
MemoryError::InvalidEntry(e) => write!(f, "invalid entry: {e}"),
MemoryError::SigningKeyRequired(e) => write!(f, "signing key required: {e}"),
}
}
}
@@ -130,6 +138,19 @@ pub struct MemoryConfig {
pub embedding_dim: usize,
pub chunk_size: usize,
pub overlap: usize,
/// Store embeddings as IEEE half precision (numpy `float16`): half the
/// bytes of the embeddings dataset on disk. Every embedding is rounded to
/// the nearest half as it enters the store, in memory as well as on disk,
/// so search results are the same before and after a reopen. Values must
/// lie within ±65504; a save outside that is `MemoryError::InvalidEntry`.
/// Fixed when the store is created (persisted in `/meta`).
///
/// **On by default for new stores**: on the full LongMemEval haystack with
/// real MiniLM embeddings every retrieval metric matched `f32`, and at
/// 100K records the file is 48% smaller (`BENCHMARKS.md`). Existing
/// stores keep the setting they were created with. Set it to `false` for
/// full-precision embeddings, e.g. for unnormalised vectors that may
/// exceed the half-precision range.
pub float16: bool,
pub compression: bool,
pub compression_level: u32,
@@ -139,6 +160,40 @@ pub struct MemoryConfig {
pub created_at: String,
pub wal_enabled: bool,
pub wal_max_entries: usize,
/// Store the vector index's own copy of the embeddings as int8 rather than
/// f32, a quarter of the memory. **On by default** for new stores.
///
/// The index's copy is the single largest part of a loaded store's
/// footprint. Quantised distances are approximate, so the candidate pool
/// is re-scored against the cache's exact embeddings before fusion, which
/// holds recall at the f32 index's level. It is also faster, not slower:
/// at equal recall, 1.63x the queries per second on x86-64 (AVX2) and
/// 1.18x on a Raspberry Pi 5 (NEON `SDOT`), with builds 1.8x and 2.3x
/// faster. See `BENCHMARKS.md`.
///
/// Persisted with the store. Stores written before this setting existed
/// have no stored value and open as `false`, so reopening an old store
/// never changes how its index is held.
///
/// Has no effect without the `hnsw` feature.
pub quantized_index: bool,
/// HNSW graph degree. Higher means a denser graph: better recall, more
/// memory and slower builds. Clamped to at least 2 when the index is
/// built, since a graph with fewer connections is not one.
///
/// Has no effect without the `hnsw` feature.
pub hnsw_m: usize,
/// Candidate list size while building the HNSW graph. Higher means a
/// better graph and a slower build; it does not affect query cost.
///
/// Has no effect without the `hnsw` feature.
pub hnsw_ef_construction: usize,
/// Candidate list size for a query, trading throughput for recall. `0`
/// keeps the default, which scales with the requested `k`
/// (`max(k * 8, 64)`) so that fusion still sees a useful pool.
///
/// Has no effect without the `hnsw` feature.
pub hnsw_ef_search: usize,
}
impl MemoryConfig {
@@ -151,7 +206,7 @@ impl MemoryConfig {
embedding_dim,
chunk_size: 512,
overlap: 50,
float16: false,
float16: true,
compression: false,
compression_level: 0,
compact_threshold: 0.3,
@@ -160,6 +215,10 @@ impl MemoryConfig {
created_at,
wal_enabled: true,
wal_max_entries: 500,
quantized_index: true,
hnsw_m: 16,
hnsw_ef_construction: 64,
hnsw_ef_search: 0,
}
}
}
@@ -258,11 +317,20 @@ pub struct HDF5Memory {
/// every record, on every single query. Built lazily on first use; see
/// [`HDF5Memory::ensure_bm25_fresh`] for how it stays in sync.
bm25: Option<bm25::BM25Index>,
/// Token filter the keyword index is built with. Changing it drops the
/// index; it is not persisted, because the index is not either.
bm25_filter: bm25::TokenFilter,
/// Activation weights changed since the last checkpoint (searches boost
/// the records they return). Cleared by `flush`.
activations_dirty: bool,
/// Opened with [`HDF5Memory::open_read_only`]: nothing may reach the disk.
read_only: bool,
/// Key that signs every checkpoint; never persisted. See
/// [`HDF5Memory::set_signing_key`].
signing_key: Option<signing::SigningKey>,
/// Checkpoints of this store are signed: the file on disk is, or a key
/// has been set. A checkpoint without a key is then refused.
signed: bool,
/// A WAL that `open()` could not read and moved aside; see
/// [`HDF5Memory::quarantined_wal`].
quarantined_wal: Option<PathBuf>,
@@ -282,7 +350,8 @@ impl HDF5Memory {
/// Create a new HDF5 memory file with the given configuration.
pub fn create(config: MemoryConfig) -> Result<Self> {
let lock = store_lock::StoreLock::acquire(&config.path)?;
let cache = MemoryCache::new(config.embedding_dim);
let mut cache = MemoryCache::new(config.embedding_dim);
cache.set_half_precision(config.float16);
let sessions = SessionCache::new();
let knowledge = KnowledgeCache::new();
@@ -314,8 +383,11 @@ impl HDF5Memory {
anomaly: anomaly::WriteAnomalyDetector::new(anomaly::AnomalyConfig::default()),
anomaly_alerts: Vec::new(),
bm25: None,
bm25_filter: bm25::TokenFilter::default(),
activations_dirty: false,
read_only: false,
signing_key: None,
signed: false,
quarantined_wal: None,
_lock: Some(lock),
})
@@ -440,7 +512,17 @@ impl HDF5Memory {
#[cfg(feature = "hnsw")]
let loaded_index = if replay_only_appended {
Self::load_vector_index(path, checkpoint.ann_generation, &cache, n_checkpoint)
Self::load_vector_index(
path,
checkpoint.ann_generation,
&cache,
n_checkpoint,
if config.quantized_index {
Storage::Int8
} else {
Storage::Float32
},
)
} else {
None
};
@@ -481,8 +563,11 @@ impl HDF5Memory {
anomaly: anomaly::WriteAnomalyDetector::new(anomaly::AnomalyConfig::default()),
anomaly_alerts: Vec::new(),
bm25: None,
bm25_filter: bm25::TokenFilter::default(),
activations_dirty: false,
read_only,
signing_key: None,
signed: checkpoint.signed,
quarantined_wal,
_lock: lock,
})
@@ -553,6 +638,7 @@ impl HDF5Memory {
generation: Option<u64>,
cache: &MemoryCache,
n_checkpoint: usize,
storage: Storage,
) -> Option<HnswIndex> {
let generation = generation?;
let bytes = std::fs::read(Self::vector_index_path(store)).ok()?;
@@ -560,15 +646,17 @@ impl HDF5Memory {
if u64::from_le_bytes(stamp.try_into().ok()?) != generation {
return None;
}
let vectors = cache.embeddings.get(..n_checkpoint)?.to_vec();
let mut index = HnswIndex::from_graph_bytes(graph, vectors).ok()?;
let vectors: Vec<Vec<f32>> = (0..n_checkpoint)
.map(|i| cache.embeddings.get(i).map(<[f32]>::to_vec))
.collect::<Option<_>>()?;
let mut index = HnswIndex::from_graph_bytes_with(graph, vectors, storage).ok()?;
if index.dimension() != cache.embedding_dim {
return None;
}
// Records appended since (replayed from the WAL) join incrementally.
for id in n_checkpoint..cache.embeddings.len() {
if cache.embeddings[id].len() != index.dimension()
|| index.insert(cache.embeddings[id].clone()) != id
|| index.insert(cache.embeddings[id].to_vec()) != id
{
return None;
}
@@ -591,7 +679,7 @@ impl HDF5Memory {
pub(crate) fn ensure_bm25_fresh(&mut self) -> &bm25::BM25Index {
let n = self.cache.chunks.len();
let bm25 = match self.bm25.take() {
Some(index) if index.len() <= n => {
Some(index) if index.len() <= n && index.token_filter() == self.bm25_filter => {
let mut index = index;
for id in index.len()..n {
if self.cache.tombstones[id] == 0 {
@@ -601,11 +689,26 @@ impl HDF5Memory {
index.pad_to(n);
index
}
_ => bm25::BM25Index::build(&self.cache.chunks, &self.cache.tombstones),
_ => bm25::BM25Index::build_with(
&self.cache.chunks,
&self.cache.tombstones,
self.bm25_filter,
),
};
self.bm25.insert(bm25)
}
/// Choose how keyword-search tokens are normalised, rebuilding the index
/// on next use. [`bm25::TokenFilter::Stemmed`] matches inflections of the
/// same word at some cost in precision; measure before adopting it (see
/// `BENCHMARKS.md`).
pub fn set_token_filter(&mut self, filter: bm25::TokenFilter) {
if filter != self.bm25_filter {
self.bm25_filter = filter;
self.bm25 = None;
}
}
/// Record `id` was tombstoned; its text is still in the cache.
fn bm25_on_delete(&mut self, id: usize) {
if let Some(index) = self.bm25.as_mut()
@@ -625,6 +728,39 @@ impl HDF5Memory {
}
}
/// Sign every checkpoint from now on with `key` (Ed25519). The key is
/// never written anywhere; set it again after every `open`. Once a store
/// is signed, a checkpoint without the key is refused
/// ([`MemoryError::SigningKeyRequired`]) rather than silently leaving it
/// unsigned. Setting a different key re-signs the store under that key
/// from the next checkpoint; a verifier trusting the old key will then
/// reject it, which is the point. Call [`AgentMemory::flush_wal`] to sign
/// right away.
pub fn set_signing_key(&mut self, key: signing::SigningKey) {
self.signing_key = Some(key);
self.signed = true;
}
/// Stop signing: the next checkpoint writes the store unsigned. The
/// deliberate way out of [`MemoryError::SigningKeyRequired`].
pub fn remove_signature(&mut self) {
self.signing_key = None;
self.signed = false;
}
/// Checkpoints of this store are signed (on disk, or from the next
/// checkpoint because a key has been set).
pub fn is_signed(&self) -> bool {
self.signed
}
/// Check the checkpoint at `path` against the public key the caller
/// trusts; see [`signing::verify_store`]. Reads the file only: it works
/// on a store another process has open.
pub fn verify(path: &Path, trusted: &signing::VerifyingKey) -> Result<signing::VerifyReport> {
signing::verify_store(path, trusted)
}
/// Flush current state to disk and truncate the WAL.
///
/// Every code path that persists the full cache to the .h5 file must
@@ -640,10 +776,28 @@ impl HDF5Memory {
// Record which WAL prefix this checkpoint contains, so a crash before
// the truncate below can't replay those entries a second time.
let wal_applied = self.wal.as_ref().map(|w| w.mark());
let signature = match &self.signing_key {
Some(key) => Some(signing::sign(
key,
&self.config,
&self.cache,
&self.sessions,
&self.knowledge,
wal_applied,
)),
None if self.signed => {
return Err(MemoryError::SigningKeyRequired(format!(
"{} is signed; set its signing key before a checkpoint \
(saves so far are held in the WAL or in memory)",
self.config.path.display()
)));
}
None => None,
};
// Written before the .h5 so a crash in between leaves a sidecar whose
// generation matches no checkpoint (ignored), never the reverse.
let ann_generation = self.persist_vector_index();
storage::write_to_disk_with_meta(
storage::write_to_disk_signed(
&self.config.path,
&self.config,
&self.cache,
@@ -652,7 +806,9 @@ impl HDF5Memory {
&schema::CheckpointMeta {
wal_applied,
ann_generation,
signed: signature.is_some(),
},
signature.as_ref(),
)?;
if let Some(ref mut w) = self.wal {
w.truncate()?;
@@ -781,6 +937,42 @@ impl HDF5Memory {
// the index length drifts from the cache length (covering any mutation path
// that doesn't call a hook, e.g. consolidation pushes).
/// Graph degree for the index, never below the 2 the builder requires:
/// a config value of 0 or 1 would otherwise panic inside `clawhdf5-ann`.
#[cfg(feature = "hnsw")]
fn hnsw_m(&self) -> usize {
self.config.hnsw_m.max(2)
}
/// Build-time candidate list size, never below the graph degree — a
/// smaller one cannot fill a node's connections.
#[cfg(feature = "hnsw")]
fn hnsw_ef_construction(&self) -> usize {
self.config.hnsw_ef_construction.max(self.hnsw_m())
}
/// Query-time candidate list size for a `k`-result search. `0` means the
/// default, which scales with `k`.
#[cfg(feature = "hnsw")]
pub(crate) fn hnsw_ef_search(&self, k: usize) -> usize {
let default = (k * 8).max(64);
if self.config.hnsw_ef_search == 0 {
default
} else {
self.config.hnsw_ef_search.max(k)
}
}
/// How the index should store its copy of the vectors, per the config.
#[cfg(feature = "hnsw")]
fn index_storage(&self) -> Storage {
if self.config.quantized_index {
Storage::Int8
} else {
Storage::Float32
}
}
/// Build an HNSW index over the entire cache, re-applying tombstones as
/// soft-deletions so node ids stay aligned with cache indices.
///
@@ -796,11 +988,15 @@ impl HDF5Memory {
if self.cache.embeddings.iter().any(|e| e.len() != dim) {
return None;
}
let mut index = HnswIndex::build_with_metric(
&self.cache.embeddings,
HNSW_M,
HNSW_EF_CONSTRUCTION,
// The index owns its vectors, so it needs rows rather than the cache's
// flat buffer. This copy is the index's own; the cache keeps one.
let rows: Vec<Vec<f32>> = self.cache.embeddings.iter().map(<[f32]>::to_vec).collect();
let mut index = HnswIndex::build_with(
&rows,
self.hnsw_m(),
self.hnsw_ef_construction(),
DistanceMetric::Cosine,
self.index_storage(),
);
for (i, &t) in self.cache.tombstones.iter().enumerate() {
if t != 0 {
@@ -826,7 +1022,7 @@ impl HDF5Memory {
let dim = index.dimension();
let appended = (self.hnsw_synced_len..n).all(|id| {
self.cache.embeddings[id].len() == dim
&& index.insert(self.cache.embeddings[id].clone()) == id
&& index.insert(self.cache.embeddings[id].to_vec()) == id
});
if appended {
for id in self.hnsw_synced_len..n {
@@ -857,7 +1053,7 @@ impl HDF5Memory {
let emb_len = self.cache.embeddings[idx].len();
match self.hnsw.as_mut() {
Some(index) if emb_len == index.dimension() => {
let id = index.insert(self.cache.embeddings[idx].clone());
let id = index.insert(self.cache.embeddings[idx].to_vec());
if id == idx {
self.hnsw_synced_len = self.cache.embeddings.len();
} else {
@@ -903,6 +1099,18 @@ impl HDF5Memory {
&self.config
}
/// The sessions recorded in this store.
pub fn sessions(&self) -> &SessionCache {
&self.sessions
}
/// Mutable access to the sessions, e.g. to add many at once. Changes
/// reach the disk at the next checkpoint (any flushing call, such as
/// [`HDF5Memory::flush_wal`] or `save_batch`), not immediately.
pub fn sessions_mut(&mut self) -> &mut SessionCache {
&mut self.sessions
}
/// Get a reference to the knowledge cache.
pub fn knowledge(&self) -> &KnowledgeCache {
&self.knowledge
@@ -965,7 +1173,29 @@ impl HDF5Memory {
/// Upsert: if an active entry with the same tags (key) exists, update it in-place.
/// Otherwise append a new entry. Use this for key-based memory stores where
/// the same key should not create duplicates.
/// A `float16` store holds embeddings as IEEE half precision, which has no
/// finite value beyond ±65504. Refuse such an embedding rather than
/// silently store infinity. (Values that are already infinite or NaN are
/// stored as they are, as in an `f32` store.)
fn check_embedding(&self, embedding: &[f32]) -> Result<()> {
if !self.config.float16 {
return Ok(());
}
let overflow = embedding
.iter()
.enumerate()
.find(|&(_, &v)| v.is_finite() && round_to_f16(v).is_infinite());
match overflow {
None => Ok(()),
Some((i, v)) => Err(MemoryError::InvalidEntry(format!(
"embedding[{i}] = {v} is outside the half-precision range (±65504) \
of this float16 store"
))),
}
}
pub fn save_or_update(&mut self, entry: MemoryEntry) -> Result<usize> {
self.check_embedding(&entry.embedding)?;
if let Some(existing_idx) = self.cache.find_by_tags(&entry.tags) {
if let Some(ref mut w) = self.wal {
let wal_entry = wal::WalEntry {
@@ -1021,6 +1251,7 @@ impl HDF5Memory {
impl AgentMemory for HDF5Memory {
fn save(&mut self, entry: MemoryEntry) -> Result<usize> {
self.check_embedding(&entry.embedding)?;
if let Some(ref mut w) = self.wal {
let wal_entry = wal::WalEntry {
entry_type: wal::WalEntryType::Save,
@@ -1062,6 +1293,10 @@ impl AgentMemory for HDF5Memory {
}
fn save_batch(&mut self, entries: Vec<MemoryEntry>) -> Result<Vec<usize>> {
// All or nothing: check every entry before storing any.
for entry in &entries {
self.check_embedding(&entry.embedding)?;
}
let mut indices = Vec::with_capacity(entries.len());
for entry in entries {
let idx = self.cache.push(
@@ -1222,6 +1457,9 @@ impl HDF5Memory {
})?;
let view = memory_strategy::CacheStoreView::new(&self.cache, &self.knowledge);
let output = strat.evaluate(&exchange, &view);
for e in &output.entries {
self.check_embedding(&e.embedding)?;
}
for e in &output.entries {
self.cache.push(
e.chunk.clone(),
@@ -1246,6 +1484,35 @@ impl HDF5Memory {
}
impl HDF5Memory {
/// Delete many records with a single checkpoint, where
/// [`AgentMemory::delete`] checkpoints once per record.
///
/// All or nothing: if any id is out of range or already deleted (or
/// repeated), nothing is deleted and `MemoryError::NotFound` is returned.
/// Unlike `delete`, this never auto-compacts, so the records stay in the
/// store as tombstones (their indices unchanged) until [`AgentMemory::compact`]
/// is called — importers use it to carry over records that were already
/// deleted in the source.
pub fn delete_batch(&mut self, ids: &[usize]) -> Result<()> {
let mut seen = std::collections::HashSet::with_capacity(ids.len());
for &id in ids {
if self.cache.tombstones.get(id).copied() != Some(0) || !seen.insert(id) {
return Err(MemoryError::NotFound(format!(
"entry {id} not found or already deleted"
)));
}
}
if ids.is_empty() {
return Ok(());
}
for &id in ids {
self.cache.mark_deleted(id);
self.hnsw_on_delete(id);
self.bm25_on_delete(id);
}
self.flush()
}
pub fn tick_session(&mut self) -> Result<()> {
let d = self.config.decay_factor;
for w in self.cache.activation_weights.iter_mut() {
@@ -1306,6 +1573,16 @@ impl HDF5Memory {
let mut promoted = 0;
for key in candidates {
// Check before taking, so a rejected entry stays in the ephemeral
// tier rather than being lost.
if let Some(emb) = self
.ephemeral
.as_ref()
.and_then(|s| s.get_entry(&key))
.and_then(|e| e.embedding.as_deref())
{
self.check_embedding(emb)?;
}
let entry = match self
.ephemeral
.as_mut()
@@ -1350,7 +1627,8 @@ impl HDF5Memory {
k: usize,
) -> Vec<SearchResult> {
// Persistent tier.
let persistent = self.hybrid_search(query_embedding, query_text, 0.7, 0.3, k);
let persistent =
self.hybrid_search_with(query_embedding, query_text, hybrid::DEFAULT_FUSION, k);
const EPHEMERAL_BOOST: f32 = 1.2;
let mut results = persistent;
@@ -1433,6 +1711,79 @@ mod tests {
}
}
#[test]
fn delete_batch_tombstones_without_compacting() {
let dir = TempDir::new().unwrap();
let path = dir.path().join("test.h5");
let mut mem = HDF5Memory::create(make_config(&dir)).unwrap();
mem.save_batch(
(0..4)
.map(|i| make_entry(&format!("record {i}"), &[i as f32, 1.0, 0.0, 0.0]))
.collect(),
)
.unwrap();
// 3 of 4 is far past compact_threshold (0.3): delete() would compact.
mem.delete_batch(&[0, 1, 3]).unwrap();
assert_eq!(mem.count(), 4);
assert_eq!(mem.count_active(), 1);
drop(mem);
let mut mem = HDF5Memory::open(&path).unwrap();
assert_eq!(mem.cache.tombstones, vec![1, 1, 0, 1]);
let hits = mem.hybrid_search(&[0.0, 1.0, 0.0, 0.0], "record", 0.5, 0.5, 10);
assert!(
hits.iter().all(|r| r.index == 2),
"tombstoned record returned"
);
}
#[test]
fn delete_batch_is_all_or_nothing() {
let dir = TempDir::new().unwrap();
let mut mem = HDF5Memory::create(make_config(&dir)).unwrap();
mem.save_batch(vec![
make_entry("a", &[1.0, 0.0, 0.0, 0.0]),
make_entry("b", &[0.0, 1.0, 0.0, 0.0]),
])
.unwrap();
for bad in [&[0, 5][..], &[1, 1][..]] {
assert!(matches!(
mem.delete_batch(bad),
Err(MemoryError::NotFound(_))
));
assert_eq!(mem.count_active(), 2, "{bad:?} deleted something");
}
mem.delete_batch(&[]).unwrap();
assert_eq!(mem.count_active(), 2);
}
#[test]
fn sessions_mut_add_at_keeps_timestamp_across_reopen() {
let dir = TempDir::new().unwrap();
let path = dir.path().join("test.h5");
let mut mem = HDF5Memory::create(make_config(&dir)).unwrap();
mem.sessions_mut()
.add_at("s-old", 2, 7, "discord", "old summary", 1.7e15);
mem.flush_wal().unwrap();
drop(mem);
let mem = HDF5Memory::open_read_only(&path).unwrap();
let s = mem.sessions();
assert_eq!(s.len(), 1);
let e = &s.entries[0];
assert_eq!(
(
e.id.as_str(),
e.start_idx,
e.end_idx,
e.channel.as_str(),
e.ts
),
("s-old", 2, 7, "discord", 1.7e15)
);
assert_eq!(s.summaries[0], "old summary");
}
#[test]
fn create_new_file() {
let dir = TempDir::new().unwrap();
@@ -1954,6 +2305,34 @@ mod tests {
assert_eq!(top_ids(&mut reopened, &q), expected_after);
}
#[test]
fn set_token_filter_rebuilds_the_keyword_index() {
let dir = TempDir::new().unwrap();
let mut mem = HDF5Memory::create(make_config(&dir)).unwrap();
mem.save(make_entry(
"I was training for a marathon",
&[1.0, 0.0, 0.0, 0.0],
))
.unwrap();
// Count only genuine keyword matches: `hybrid_search` also returns
// zero-score filler when fewer than k records are relevant.
let hits = |mem: &mut HDF5Memory| {
mem.hybrid_search(&[0.0, 0.0, 0.0, 0.0], "trains", 0.0, 1.0, 5)
.iter()
.filter(|r| r.score > 0.0)
.count()
};
assert_eq!(hits(&mut mem), 0);
mem.set_token_filter(bm25::TokenFilter::Stemmed);
assert_eq!(hits(&mut mem), 1, "index should have been rebuilt stemmed");
// And back, rebuilding again.
mem.set_token_filter(bm25::TokenFilter::Plain);
assert_eq!(hits(&mut mem), 0);
}
#[test]
fn keyword_index_stays_in_sync_through_every_mutation() {
let dir = TempDir::new().unwrap();
crates/clawhdf5-agent/src/openclaw.rs
+30 -66
View File
@@ -1,10 +1,12 @@
//! OpenClaw Integration Layer.
//! A Markdown-oriented memory backend over [`crate::HDF5Memory`].
//!
//! Bridge between OpenClaw agent gateway (Markdown + sqlite-vec) and the
//! clawhdf5 HDF5-backed memory backend. Provides:
//! Named for OpenClaw, whose workspace memory is Markdown, but **not an
//! OpenClaw plugin**: nothing here registers with OpenClaw, and the
//! integration it was written for never worked (see `docs/openclaw.md`).
//! Provides:
//!
//! - [`MemoryBackend`] — the trait OpenClaw implements against.
//! - [`ClawhdfBackend`] — concrete HDF5-backed implementation.
//! - [`MemoryBackend`] — search / read back / write / ingest / export.
//! - [`ClawhdfBackend`] — the HDF5-backed implementation.
//! - [`MarkdownParser`] — splits Markdown into [`MarkdownSection`] records.
//! - [`MarkdownExporter`] — renders sections back to Markdown text.
@@ -13,9 +15,8 @@ use std::path::{Path, PathBuf};
use std::time::{SystemTime, UNIX_EPOCH};
use crate::{
AgentMemory, HDF5Memory, MemoryConfig, MemoryEntry,
confidence::{ConfidenceConfig, ScoredResult, reject_low_confidence},
reranker::{ReRankConfig, RerankInput, rerank},
AgentMemory, HDF5Memory, MemoryConfig, MemoryEntry, SearchOptions,
confidence::ConfidenceConfig, reranker::ReRankConfig,
};
// ─────────────────────────────────────────────────────────────────────────────
@@ -62,7 +63,8 @@ pub struct BackendStats {
// MemoryBackend trait
// ─────────────────────────────────────────────────────────────────────────────
/// Interface that OpenClaw uses to interact with a memory backend.
/// A Markdown-oriented memory backend: search, read back by path, write,
/// ingest and export.
///
/// Implementors provide persistent storage, full-text + vector search,
/// Markdown ingestion / export, and statistics.
@@ -319,7 +321,7 @@ impl MarkdownExporter {
///
/// # Path mapping
///
/// OpenClaw addresses memories by file path (e.g. `"memory/user.md"`).
/// Memories are addressed by file path (e.g. `"memory/user.md"`).
/// Internally every [`MemoryEntry`] stores the originating path as its
/// `source_channel`. Section sub-paths are stored as
/// `"<path>::<heading>"`.
@@ -422,7 +424,7 @@ impl ClawhdfBackend {
// ── Compaction & Consolidation hooks (7.6) ────────────────────────────
/// Run a compaction cycle — called by OpenClaw during session compaction.
/// Run a compaction cycle (decay, compaction, WAL flush).
///
/// Sequence:
/// 1. `tick_session()` — apply Hebbian decay to all activation weights.
@@ -466,7 +468,7 @@ impl ClawhdfBackend {
let record = MemoryRecord {
id: i as u64,
chunk: cache.chunks[i].clone(),
embedding: cache.embeddings[i].clone(),
embedding: cache.embeddings[i].to_vec(),
tier: MemoryTier::Working,
importance: cache.activation_weights[i],
access_count: 0,
@@ -524,67 +526,27 @@ impl ClawhdfBackend {
impl MemoryBackend for ClawhdfBackend {
/// Search using hybrid vector + BM25 retrieval, then re-rank and
/// confidence-filter.
/// confidence-filter — [`HDF5Memory::search`] with both stages on.
fn search(
&mut self,
query_text: &str,
query_embedding: &[f32],
k: usize,
) -> Vec<MemorySearchResult> {
// 1. Hybrid retrieval (RRF-blended vector + BM25).
let candidates = k.saturating_mul(3).max(10);
let raw = self
.memory
.hybrid_search(query_embedding, query_text, 0.7, 0.3, candidates);
if raw.is_empty() {
return Vec::new();
}
let now = Self::now_secs();
// 2. Re-rank using temporal recency, source authority, Hebbian weight.
let rerank_inputs: Vec<RerankInput> = raw
.iter()
.map(|r| RerankInput {
index: r.index,
timestamp: r.timestamp,
source_channel: r.source_channel.clone(),
raw_activation: r.activation,
})
.collect();
let reranked = rerank(&rerank_inputs, &self.rerank_config, now);
// 3. Confidence rejection.
let scored: Vec<ScoredResult> = reranked
.iter()
.map(|r| ScoredResult {
index: r.index,
score: r.combined_score,
})
.collect();
let confident = reject_low_confidence(&scored, &self.confidence_config);
// 4. Map back to MemorySearchResult; preserve raw text via index lookup.
let raw_by_idx: HashMap<usize, &crate::SearchResult> =
raw.iter().map(|r| (r.index, r)).collect();
confident
let options = SearchOptions::new(k)
.with_rerank(self.rerank_config)
.with_confidence(self.confidence_config.clone())
.at_time(Self::now_secs());
self.memory
.search(query_embedding, query_text, &options)
.into_iter()
.take(k)
.filter_map(|sr| {
let r = raw_by_idx.get(&sr.index)?;
let path = r.source_channel.clone();
Some(MemorySearchResult {
text: r.chunk.clone(),
score: sr.score,
path: path.clone(),
.map(|r| MemorySearchResult {
text: r.chunk,
score: r.score,
path: r.source_channel.clone(),
line_range: None,
timestamp: Some(r.timestamp),
source: path,
})
source: r.source_channel,
})
.collect()
}
@@ -713,11 +675,13 @@ impl MemoryBackend for ClawhdfBackend {
let total_records = cache.count_active();
// A record saved without an embedding occupies a zero row, so "has an
// embedding" is "has a non-zero norm" rather than "row is non-empty".
let total_embeddings = cache
.embeddings
.norms
.iter()
.enumerate()
.filter(|(i, emb)| cache.tombstones[*i] == 0 && !emb.is_empty())
.filter(|(i, norm)| cache.tombstones[*i] == 0 && **norm > 0.0)
.count();
let file_size_bytes = std::fs::metadata(&self.hdf5_path)
crates/clawhdf5-agent/src/provenance.rs
+1 -1
View File
@@ -2,7 +2,7 @@
//!
//! Records the origin, authorship, and a content hash of every memory chunk
//! so the system can detect *accidental* corruption and trace data lineage.
//! The hash is unkeyed (see [`fnv1a_64`]) — this is not a tamper-evidence or
//! The hash is unkeyed (FNV-1a) — this is not a tamper-evidence or
//! authenticity guarantee.
use std::collections::HashMap;
crates/clawhdf5-agent/src/query_expand.rs
+162 -8
View File
@@ -6,6 +6,11 @@
//! - Temporal expansion (time-related rewrites)
//! - Morphological variants (stemming-like transforms)
//! - Knowledge graph expansion (entity aliases and neighbors)
//!
//! The morphological rules are crude suffix swaps, so some variants are not
//! words ("during" -> "dured"). That is tolerable for a BM25 stage, which
//! simply finds no postings for a nonsense term, but it means expansion is not
//! free: measure before enabling it on a retrieval path.
use crate::knowledge::KnowledgeCache;
@@ -340,18 +345,85 @@ fn contains_phrase(text: &str, phrase: &str) -> bool {
/// Replace a phrase in `text` case-insensitively, preserving surrounding case.
fn replace_word_case_insensitive(text: &str, from: &str, to: &str) -> String {
case_insensitive_replace(text, from, to)
replace_first(text, from, to, MatchKind::WholeWord)
}
fn case_insensitive_replace(text: &str, from: &str, to: &str) -> String {
let lower = text.to_lowercase();
let lower_from = from.to_lowercase();
if let Some(pos) = lower.find(&lower_from) {
let end = pos + from.len();
format!("{}{}{}", &text[..pos], to, &text[end..])
} else {
text.to_string()
replace_first(text, from, to, MatchKind::Substring)
}
/// Whether a match may fall inside a larger word.
#[derive(Clone, Copy, PartialEq)]
enum MatchKind {
/// Match anywhere, including inside another word.
Substring,
/// Match only when both ends sit on a word boundary.
WholeWord,
}
/// Replace the first case-insensitive match of `from` in `text` with `to`.
///
/// Matching walks the *original* string rather than a lowercased copy. The
/// previous implementation searched `text.to_lowercase()` and then sliced
/// `text` with the offsets it found, which only holds while lowercasing
/// preserves byte length. It does not: Turkish `İ` (2 bytes) lowercases to
/// `i` + U+0307 (3 bytes), so every later offset was wrong — silently
/// corrupting the output, or panicking when an offset landed inside a
/// character or past the end. `"İ AI"` was enough to panic.
fn replace_first(text: &str, from: &str, to: &str, kind: MatchKind) -> String {
match find_case_insensitive(text, from, kind) {
Some((start, end)) => {
let mut out = String::with_capacity(text.len() - (end - start) + to.len());
out.push_str(&text[..start]);
out.push_str(to);
out.push_str(&text[end..]);
out
}
None => text.to_string(),
}
}
/// Byte range of the first case-insensitive match of `needle` in `haystack`.
fn find_case_insensitive(haystack: &str, needle: &str, kind: MatchKind) -> Option<(usize, usize)> {
if needle.is_empty() {
return None;
}
let lowered: Vec<char> = needle.chars().flat_map(char::to_lowercase).collect();
let is_word = |c: char| c.is_alphanumeric() || c == '_';
for (start, _) in haystack.char_indices() {
if kind == MatchKind::WholeWord
&& haystack[..start].chars().next_back().is_some_and(is_word)
{
continue; // mid-word: "ai" inside "training"
}
let mut matched = 0usize;
let mut end = start;
for (offset, ch) in haystack[start..].char_indices() {
if matched == lowered.len() {
break;
}
let mut consumed_all = true;
for lc in ch.to_lowercase() {
if lowered.get(matched) != Some(&lc) {
consumed_all = false;
break;
}
matched += 1;
}
if !consumed_all {
break;
}
end = start + offset + ch.len_utf8();
}
if matched == lowered.len()
&& !(kind == MatchKind::WholeWord
&& haystack[end..].chars().next().is_some_and(is_word))
{
return Some((start, end));
}
}
None
}
/// Simple whitespace/punctuation tokenizer.
@@ -637,4 +709,86 @@ mod tests {
expanded.iter().map(|x| &x.text).collect::<Vec<_>>()
);
}
#[test]
fn acronyms_only_match_whole_words() {
let ex = QueryExpander::new(QueryExpansionConfig::default());
// "training" contains "ai", "programming" contains "pr". These used to
// be rewritten to "trArtificial Intelligencening" and
// "Pull Requestogramming".
for query in [
"How many miles during my marathon training?",
"Which programming language did I pick?",
"I updated the maintainer list",
] {
for expansion in ex.expand(query) {
assert!(
expansion.expansion_type != "acronym",
"{query:?} produced {expansion:?}"
);
}
}
// A real acronym still expands, in both directions.
let texts: Vec<String> = ex
.expand("What about the API and the database?")
.into_iter()
.filter(|e| e.expansion_type == "acronym")
.map(|e| e.text)
.collect();
assert!(
texts
.iter()
.any(|t| t.contains("Application Programming Interface")),
"{texts:?}"
);
assert!(texts.iter().any(|t| t.contains("DB")), "{texts:?}");
}
#[test]
fn non_ascii_queries_do_not_panic_or_corrupt() {
let ex = QueryExpander::new(QueryExpansionConfig::default());
// Turkish 'İ' is 2 bytes but lowercases to 3, so offsets taken from a
// lowercased copy no longer line up with the original. `"İ AI"` used
// to panic; `"İstanbul AI trip"` used to silently eat a character.
for query in ["İ AI", "İé AI", "İİ ML", "İstanbul AI trip", "ǰ ML notes"] {
for expansion in ex.expand(query) {
assert!(
expansion.text.contains('İ') || expansion.text.contains('ǰ'),
"{query:?} lost its leading character: {expansion:?}"
);
}
}
let expanded = ex.expand("İstanbul AI trip");
assert!(
expanded
.iter()
.any(|e| e.text == "İstanbul Artificial Intelligence trip"),
"{expanded:?}"
);
}
#[test]
fn whole_word_matching_handles_string_edges_and_case() {
assert_eq!(
replace_word_case_insensitive("ai tools", "AI", "Artificial Intelligence"),
"Artificial Intelligence tools"
);
assert_eq!(
replace_word_case_insensitive("tools for ai", "AI", "Artificial Intelligence"),
"tools for Artificial Intelligence"
);
assert_eq!(
replace_word_case_insensitive("the aim", "AI", "Artificial Intelligence"),
"the aim",
"must not match inside a word"
);
assert_eq!(
replace_word_case_insensitive("no match here", "xyz", "abc"),
"no match here"
);
// Only the first occurrence is replaced, as before.
assert_eq!(
replace_word_case_insensitive("ai and ai", "ai", "ML"),
"ML and ai"
);
}
}
crates/clawhdf5-agent/src/reranker.rs
+51 -2
View File
@@ -4,8 +4,10 @@
//! into a single composite score for each retrieved result.
/// Configuration for the multi-factor re-ranker.
#[derive(Debug, Clone)]
#[derive(Debug, Clone, Copy)]
pub struct ReRankConfig {
/// Weight applied to the retrieval score the candidate arrived with.
pub relevance_weight: f32,
/// Weight applied to the temporal decay score (0.0–1.0).
pub temporal_weight: f32,
/// Weight applied to the source authority score (0.0–1.0).
@@ -20,6 +22,9 @@ pub struct ReRankConfig {
impl Default for ReRankConfig {
fn default() -> Self {
Self {
// Relevance leads: the metadata signals break ties and nudge, they
// do not decide. See `BENCHMARKS.md`, "Recency discrimination".
relevance_weight: 1.0,
temporal_weight: 0.3,
authority_weight: 0.2,
activation_weight: 0.5,
@@ -41,6 +46,8 @@ pub struct ReRankResult {
pub authority_score: f32,
/// Normalised Hebbian activation score in [0, 1].
pub activation_score: f32,
/// The retrieval score carried through from the input.
pub relevance_score: f32,
}
/// Compute an exponential decay temporal score.
@@ -105,6 +112,15 @@ pub struct RerankInput {
pub source_channel: String,
/// Raw Hebbian activation weight for this entry.
pub raw_activation: f32,
/// The retrieval score that put this entry in the candidate list.
///
/// Re-ranking is meant to *adjust* the retriever's ordering with signals
/// it does not have, not to replace it. Without this the combined score
/// was made of recency, authority and activation alone, so a candidate
/// pool came back ordered by age with its relevance ordering discarded.
/// Callers with no meaningful score can pass the same value for every
/// entry, which reduces to the old behaviour.
pub relevance: f32,
}
/// Re-rank a list of retrieval results using multi-factor scoring.
@@ -138,7 +154,8 @@ pub fn rerank(
let auth = source_authority_score(&inp.source_channel);
let act = activation_score(inp.raw_activation);
let combined = config.temporal_weight * ts
let combined = config.relevance_weight * inp.relevance
+ config.temporal_weight * ts
+ config.authority_weight * auth
+ config.activation_weight * act;
@@ -148,6 +165,7 @@ pub fn rerank(
temporal_score: ts,
authority_score: auth,
activation_score: act,
relevance_score: inp.relevance,
}
})
.collect();
@@ -253,22 +271,51 @@ mod tests {
timestamp: 0.0, // very old
source_channel: "other".to_string(),
raw_activation: 0.1,
relevance: 0.0,
},
RerankInput {
index: 1,
timestamp: 86_400.0, // one day ago
source_channel: "conversation".to_string(),
raw_activation: 0.5,
relevance: 0.0,
},
RerankInput {
index: 2,
timestamp: 172_800.0, // "now"
source_channel: "user_correction".to_string(),
raw_activation: 1.0,
relevance: 0.0,
},
]
}
#[test]
fn relevance_leads_but_recency_breaks_near_ties() {
let entry = |index, timestamp, relevance| RerankInput {
index,
timestamp,
source_channel: "conversation".to_string(),
raw_activation: 1.0,
relevance,
};
let now = 10.0 * 86_400.0;
let config = ReRankConfig::default();
// A clearly better match wins despite being much older. Before
// `relevance` existed the combined score ignored it entirely, so this
// returned the newer, irrelevant entry.
let ranked = rerank(&[entry(0, 0.0, 1.0), entry(1, now, 0.1)], &config, now);
assert_eq!(ranked[0].index, 0, "{ranked:?}");
// Between near-equal matches, the newer one wins.
let ranked = rerank(&[entry(0, 0.0, 0.80), entry(1, now, 0.79)], &config, now);
assert_eq!(ranked[0].index, 1, "{ranked:?}");
// The breakdown carries the relevance through.
assert_eq!(ranked[0].relevance_score, 0.79);
}
#[test]
fn rerank_returns_all_entries() {
let inputs = make_inputs();
@@ -302,6 +349,7 @@ mod tests {
#[test]
fn rerank_score_breakdown_matches_manual_calculation() {
let config = ReRankConfig {
relevance_weight: 0.0,
temporal_weight: 1.0,
authority_weight: 0.0,
activation_weight: 0.0,
@@ -312,6 +360,7 @@ mod tests {
timestamp: 0.0,
source_channel: "other".to_string(),
raw_activation: 0.5,
relevance: 0.0,
}];
let now = 3600.0_f64; // exactly one half-life later
let results = rerank(&inputs, &config, now);
crates/clawhdf5-agent/src/schema.rs
+196 -26
View File
@@ -15,6 +15,9 @@ use crate::session::SessionCache;
use crate::wal::WalMark;
pub const SCHEMA_VERSION: &str = "1.0";
/// Writer-version tag stored in `/meta` as `edgehdf5_version`. Kept for file
/// compatibility; despite the name it has nothing to do with ZeroClaw, which
/// does not use clawhdf5.
pub const ZEROCLAW_VERSION: &str = "0.8.0";
/// `/meta` attributes holding the [`WalMark`] of the WAL prefix already folded
@@ -23,6 +26,7 @@ pub const ZEROCLAW_VERSION: &str = "0.8.0";
const WAL_APPLIED_LEN_ATTR: &str = "wal_applied_len";
const WAL_APPLIED_CRC_ATTR: &str = "wal_applied_crc";
const ANN_GENERATION_ATTR: &str = "ann_generation";
const SIG_VERSION_ATTR: &str = "sig_version";
/// Build a complete HDF5 file from the in-memory state.
pub fn build_hdf5_file(
@@ -46,7 +50,7 @@ pub fn build_hdf5_file_with_mark(
) -> Result<Vec<u8>, MemoryError> {
let meta = CheckpointMeta {
wal_applied,
ann_generation: None,
..CheckpointMeta::default()
};
build_hdf5_file_with_meta(config, cache, sessions, knowledge, &meta)
}
@@ -61,6 +65,10 @@ pub struct CheckpointMeta {
/// one left over from another checkpoint can never be attached to records
/// it wasn't built from.
pub ann_generation: Option<u64>,
/// The checkpoint carries an Ed25519 signature (see [`crate::signing`]).
/// Read-only: whether a checkpoint is *written* signed is decided by the
/// signature passed to [`build_hdf5_file_signed`].
pub signed: bool,
}
/// [`build_hdf5_file`] with checkpoint bookkeeping.
@@ -70,6 +78,19 @@ pub fn build_hdf5_file_with_meta(
sessions: &SessionCache,
knowledge: &KnowledgeCache,
checkpoint: &CheckpointMeta,
) -> Result<Vec<u8>, MemoryError> {
build_hdf5_file_signed(config, cache, sessions, knowledge, checkpoint, None)
}
/// [`build_hdf5_file_with_meta`], plus a signed manifest of the contents
/// (see [`crate::signing`]).
pub fn build_hdf5_file_signed(
config: &MemoryConfig,
cache: &MemoryCache,
sessions: &SessionCache,
knowledge: &KnowledgeCache,
checkpoint: &CheckpointMeta,
signature: Option<&crate::signing::StoredSignature>,
) -> Result<Vec<u8>, MemoryError> {
let wal_applied = checkpoint.wal_applied;
let mut builder = clawhdf5::FileBuilder::new();
@@ -104,6 +125,19 @@ pub fn build_hdf5_file_with_meta(
"wal_max_entries",
AttrValue::I64(config.wal_max_entries as i64),
);
meta.set_attr(
"quantized_index",
AttrValue::I64(config.quantized_index.into()),
);
meta.set_attr("hnsw_m", AttrValue::I64(config.hnsw_m as i64));
meta.set_attr(
"hnsw_ef_construction",
AttrValue::I64(config.hnsw_ef_construction as i64),
);
meta.set_attr(
"hnsw_ef_search",
AttrValue::I64(config.hnsw_ef_search as i64),
);
meta.set_attr(
"edgehdf5_version",
AttrValue::String(ZEROCLAW_VERSION.into()),
@@ -117,11 +151,42 @@ pub fn build_hdf5_file_with_meta(
// round trip through every reader.
meta.set_attr(ANN_GENERATION_ATTR, AttrValue::I64(generation as i64));
}
if let Some(sig) = signature {
use crate::signing::to_hex;
let m = &sig.manifest;
meta.set_attr(
SIG_VERSION_ATTR,
AttrValue::I64(crate::signing::MANIFEST_VERSION),
);
meta.set_attr("sig_algorithm", AttrValue::String("ed25519".into()));
meta.set_attr("sig_public_key", AttrValue::String(to_hex(&sig.public_key)));
meta.set_attr("sig_signature", AttrValue::String(to_hex(&sig.signature)));
meta.set_attr("sig_record_count", AttrValue::I64(m.record_count as i64));
meta.set_attr(
"sig_records_root",
AttrValue::String(to_hex(&m.records_root)),
);
meta.set_attr("sig_settings", AttrValue::String(to_hex(&m.settings)));
meta.set_attr("sig_sessions", AttrValue::String(to_hex(&m.sessions)));
meta.set_attr("sig_graph", AttrValue::String(to_hex(&m.graph)));
}
// Need at least one dataset in the group for it to be a proper group
meta.create_dataset("_marker").with_u8_data(&[1]).compact();
let finished_meta = meta.finish();
builder.add_group(finished_meta);
// /integrity: the signed per-record hashes, so verification can say
// which records changed.
if let Some(sig) = signature {
let mut group = builder.create_group("integrity");
let flat: Vec<u8> = sig.record_hashes.iter().flatten().copied().collect();
group
.create_dataset("record_hashes")
.with_u8_data(&flat)
.with_shape(&[sig.record_hashes.len() as u64, 32]);
builder.add_group(group.finish());
}
// /memory group
build_memory_group(&mut builder, config, cache)?;
@@ -146,20 +211,27 @@ fn build_memory_group(
// chunks: fixed-length string array
write_string_dataset(&mut group, "chunks", &cache.chunks);
// embeddings: f32 [N x D]
// embeddings: [N x D], f32 — or IEEE half precision for a `float16`
// store. The cache already holds half-rounded values then, so this
// conversion is exact and a reopened store sees the same numbers.
let n = cache.embeddings.len() as u64;
let d = cache.embedding_dim as u64;
let flat = cache.flat_embeddings();
{
let ds = group
.create_dataset("embeddings")
.with_f32_data(&flat)
.with_shape(&[n, d]);
let ds = group.create_dataset("embeddings");
let elem_bytes: u64 = if config.float16 {
ds.with_f16_data(flat);
2
} else {
ds.with_f32_data(flat);
4
};
ds.with_shape(&[n, d]);
// Chunk size tuning: target ~256KB per chunk for optimal I/O
if n > 0 && d > 0 {
let target_chunk_bytes: u64 = 256 * 1024;
let rows_per_chunk = (target_chunk_bytes / (d * 4)).max(1).min(n);
let rows_per_chunk = (target_chunk_bytes / (d * elem_bytes)).max(1).min(n);
ds.with_chunks(&[rows_per_chunk, d]);
// Compression. Shuffle is applied automatically (auto-shuffle
@@ -405,10 +477,34 @@ fn write_string_dataset(
}
}
/// `/meta`'s attributes, failing if any of them cannot be read.
///
/// `Group::attrs` leaves out an attribute it cannot decode. For the store's
/// settings that would silently fall back to defaults (e.g. `float16`, the
/// WAL mark), so an unreadable attribute is an error here, as it was before
/// `attrs` became tolerant.
fn meta_attrs(
file: &clawhdf5::File,
) -> Result<std::collections::HashMap<String, AttrValue>, MemoryError> {
let meta = file
.group("meta")
.map_err(|e| MemoryError::Schema(format!("missing /meta group: {e}")))?;
let (attrs, errors) = meta
.attrs_with_errors()
.map_err(|e| MemoryError::Schema(format!("cannot read /meta attrs: {e}")))?;
if let Some(e) = errors.first() {
return Err(MemoryError::Schema(format!(
"cannot read /meta attrs: {} unreadable, first: {e}",
errors.len()
)));
}
Ok(attrs)
}
/// Validate an HDF5 file has the correct schema and load all data.
/// Read the checkpoint's [`WalMark`] from `/meta`, if it has one.
pub fn read_wal_mark(file: &clawhdf5::File) -> Option<WalMark> {
let attrs = file.group("meta").ok()?.attrs().ok()?;
let attrs = meta_attrs(file).ok()?;
let len = match attrs.get(WAL_APPLIED_LEN_ATTR)? {
AttrValue::I64(v) => u64::try_from(*v).ok()?,
_ => return None,
@@ -420,19 +516,75 @@ pub fn read_wal_mark(file: &clawhdf5::File) -> Option<WalMark> {
Some(WalMark { len, crc })
}
/// Read a checkpoint's signature, if it has one. A signature whose
/// attributes are present but malformed is an error, not "unsigned".
pub fn read_signature(
file: &clawhdf5::File,
) -> Result<Option<crate::signing::StoredSignature>, MemoryError> {
use crate::signing::{Manifest, StoredSignature, from_hex};
let attrs = meta_attrs(file)?;
let version = match attrs.get(SIG_VERSION_ATTR) {
None => return Ok(None),
Some(AttrValue::I64(v)) => *v,
Some(_) => return Err(MemoryError::Schema("malformed sig_version".into())),
};
if version != crate::signing::MANIFEST_VERSION {
return Err(MemoryError::Schema(format!(
"unsupported signature version {version}"
)));
}
fn hex<const N: usize>(
attrs: &std::collections::HashMap<String, AttrValue>,
name: &str,
) -> Result<[u8; N], MemoryError> {
match attrs.get(name) {
Some(AttrValue::String(s)) => from_hex::<N>(s),
_ => None,
}
.ok_or_else(|| MemoryError::Schema(format!("malformed or missing {name}")))
}
let record_count = match attrs.get("sig_record_count") {
Some(AttrValue::I64(v)) if *v >= 0 => *v as u64,
_ => return Err(MemoryError::Schema("malformed sig_record_count".into())),
};
let group = file
.group("integrity")
.map_err(|e| MemoryError::Schema(format!("signed checkpoint without /integrity: {e}")))?;
let flat = read_u8_dataset(&group, "record_hashes")?;
if flat.len() % 32 != 0 {
return Err(MemoryError::Schema(
"/integrity/record_hashes is not a whole number of hashes".into(),
));
}
let record_hashes = flat.as_chunks::<32>().0.to_vec();
Ok(Some(StoredSignature {
manifest: Manifest {
record_count,
records_root: hex::<32>(&attrs, "sig_records_root")?,
settings: hex::<32>(&attrs, "sig_settings")?,
sessions: hex::<32>(&attrs, "sig_sessions")?,
graph: hex::<32>(&attrs, "sig_graph")?,
},
record_hashes,
public_key: hex::<32>(&attrs, "sig_public_key")?,
signature: hex::<64>(&attrs, "sig_signature")?,
}))
}
/// Read the checkpoint bookkeeping from `/meta`.
pub fn read_checkpoint_meta(file: &clawhdf5::File) -> CheckpointMeta {
let ann_generation = file
.group("meta")
let ann_generation =
meta_attrs(file)
.ok()
.and_then(|g| g.attrs().ok())
.and_then(|attrs| match attrs.get(ANN_GENERATION_ATTR) {
Some(AttrValue::I64(v)) => Some(*v as u64),
_ => None,
});
let signed = meta_attrs(file).is_ok_and(|attrs| attrs.contains_key(SIG_VERSION_ATTR));
CheckpointMeta {
wal_applied: read_wal_mark(file),
ann_generation,
signed,
}
}
@@ -440,12 +592,7 @@ pub fn validate_and_load(
file: &clawhdf5::File,
) -> Result<(MemoryConfig, MemoryCache, SessionCache, KnowledgeCache), MemoryError> {
// Read /meta group attributes
let meta = file
.group("meta")
.map_err(|e| MemoryError::Schema(format!("missing /meta group: {e}")))?;
let attrs = meta
.attrs()
.map_err(|e| MemoryError::Schema(format!("cannot read /meta attrs: {e}")))?;
let attrs = meta_attrs(file)?;
let schema_version = match attrs.get("schema_version") {
Some(AttrValue::String(s)) => s.clone(),
@@ -484,10 +631,32 @@ pub fn validate_and_load(
wal_max_entries: optional_i64_attr(&attrs, "wal_max_entries")
.and_then(|v| usize::try_from(v).ok())
.unwrap_or(500),
// `false`, not the new-store default: a store written before this
// setting existed was built with an f32 index, and reopening it must
// not silently change that.
quantized_index: optional_bool_attr(&attrs, "quantized_index", false),
hnsw_m: optional_i64_attr(&attrs, "hnsw_m")
.and_then(|v| usize::try_from(v).ok())
.unwrap_or(16),
hnsw_ef_construction: optional_i64_attr(&attrs, "hnsw_ef_construction")
.and_then(|v| usize::try_from(v).ok())
.unwrap_or(64),
hnsw_ef_search: optional_i64_attr(&attrs, "hnsw_ef_search")
.and_then(|v| usize::try_from(v).ok())
.unwrap_or(0),
};
// Load /memory group
let memory_cache = load_memory_group(file, embedding_dim)?;
let mut memory_cache = load_memory_group(file, embedding_dim)?;
// A float16 store's cache holds half-rounded embeddings. Embeddings read
// from an f16 dataset already are; a float16 store whose last checkpoint
// predates half-precision storage is still f32 on disk and is rounded
// here.
if config.float16 && embeddings_are_f16(file) {
memory_cache.half_precision = true;
} else {
memory_cache.set_half_precision(config.float16);
}
// Load /sessions group
let session_cache = load_sessions_group(file)?;
@@ -563,12 +732,7 @@ fn load_memory_group(
.collect(),
};
// Unflatten embeddings
let embeddings: Vec<Vec<f32>> = flat_embeddings
.chunks(embedding_dim)
.map(|c| c.to_vec())
.collect();
// No unflattening: the cache stores the buffer as it is on disk.
// Read activation_weights if present, default to vec![1.0; N] for backward compat
let activation_weights = match read_f32_dataset(&group, "activation_weights") {
Ok(w) if w.len() == n => w,
@@ -576,7 +740,7 @@ fn load_memory_group(
};
cache.chunks = chunks;
cache.embeddings = embeddings;
cache.embeddings.set_flat(embedding_dim, flat_embeddings);
cache.source_channels = source_channels;
cache.timestamps = timestamps;
cache.session_ids = session_ids;
@@ -584,7 +748,6 @@ fn load_memory_group(
cache.tombstones = tombstones;
cache.norms = norms;
cache.activation_weights = activation_weights;
cache.rebuild_flat();
Ok(cache)
}
@@ -736,6 +899,13 @@ fn read_string_dataset_from_group(
.map_err(|e| MemoryError::Hdf5(format!("cannot read strings from {name}: {e}")))
}
/// Whether `/memory/embeddings` is stored as IEEE half precision.
fn embeddings_are_f16(file: &clawhdf5::File) -> bool {
file.dataset("memory/embeddings")
.and_then(|ds| ds.dtype())
.is_ok_and(|dt| matches!(dt, clawhdf5::DType::Other(ref s) if s == "float16"))
}
fn read_f32_dataset(group: &clawhdf5::Group<'_>, name: &str) -> Result<Vec<f32>, MemoryError> {
let ds = group
.dataset(name)
crates/clawhdf5-agent/src/search.rs
+305 -29
View File
@@ -2,61 +2,196 @@
use std::path::Path;
use std::collections::HashSet;
use crate::bm25;
use crate::confidence::{ConfidenceConfig, ScoredResult, reject_low_confidence};
use crate::hybrid;
use crate::reranker::{ReRankConfig, RerankInput, rerank};
use crate::{HDF5Memory, MAX_ACTIVATION_WEIGHT, MemoryError, Result, SearchResult};
/// Options for [`HDF5Memory::search`].
///
/// [`SearchOptions::new`] is plain hybrid search with the tuned default
/// fusion — the same as `hybrid_search_with(.., hybrid::DEFAULT_FUSION, k)`.
/// Every stage beyond that is opt-in.
#[derive(Debug, Clone)]
pub struct SearchOptions {
/// Number of results to return.
pub k: usize,
/// How the vector and keyword stages are combined.
pub fusion: hybrid::Fusion,
/// Only consider records whose `source_channel` is one of these. The
/// filter applies *before* ranking, so a filtered search still returns up
/// to `k` results and scores are normalised over the records it can
/// return. `None` searches everything; an empty list matches nothing.
pub source_channels: Option<Vec<String>>,
/// Re-rank a candidate pool by retrieval relevance, recency, source
/// authority and activation — the pipeline the OpenClaw backend runs.
pub rerank: Option<ReRankConfig>,
/// Candidates retrieved for re-ranking; 0 means `max(3k, 10)`.
pub rerank_pool: usize,
/// Drop low-confidence results (after re-ranking, when that is on).
pub confidence: Option<ConfidenceConfig>,
/// The time recency is measured from, in seconds since the epoch.
/// `None` uses the system clock.
pub now: Option<f64>,
}
impl SearchOptions {
pub fn new(k: usize) -> Self {
Self {
k,
fusion: hybrid::DEFAULT_FUSION,
source_channels: None,
rerank: None,
rerank_pool: 0,
confidence: None,
now: None,
}
}
pub fn with_fusion(mut self, fusion: hybrid::Fusion) -> Self {
self.fusion = fusion;
self
}
/// Search only records from these source channels.
pub fn with_sources<S: Into<String>>(mut self, channels: impl IntoIterator<Item = S>) -> Self {
self.source_channels = Some(channels.into_iter().map(Into::into).collect());
self
}
pub fn with_rerank(mut self, config: ReRankConfig) -> Self {
self.rerank = Some(config);
self
}
pub fn with_confidence(mut self, config: ConfidenceConfig) -> Self {
self.confidence = Some(config);
self
}
/// Measure recency from `now` (seconds since the epoch) instead of the
/// system clock — for reproducible results and tests.
pub fn at_time(mut self, now: f64) -> Self {
self.now = Some(now);
self
}
}
impl Default for SearchOptions {
fn default() -> Self {
Self::new(10)
}
}
impl HDF5Memory {
/// Vector + keyword scoring stage of [`HDF5Memory::hybrid_search`].
/// Vector + keyword scoring stage of [`HDF5Memory::search`].
///
/// Without the `hnsw` feature this is a full linear cosine scan (the exact
/// previous behaviour, also used as the correctness oracle in tests). With
/// `hnsw` enabled and an index available, the vector candidates come from an
/// approximate-nearest-neighbour search over an over-fetched pool, then merge
/// with BM25 via the shared [`hybrid::merge_vector_keyword`].
///
/// `exclude`, when given, marks records that must not be returned (1 =
/// excluded; it covers tombstones too). The index is over-fetched in
/// proportion to how much the mask removes. Surfacing `pool` candidates
/// costs the index roughly `pool × M` distance evaluations, while an exact
/// scan of the allowed records costs one each — so whenever that scan is
/// the cheaper of the two it is used instead, and it is also the fallback
/// if the pool comes back with too few allowed hits (the allowed records
/// sit away from the query). A filtered search never comes back short.
#[cfg(feature = "hnsw")]
fn vector_keyword_search(
&mut self,
query_embedding: &[f32],
query_text: &str,
bm25: &bm25::BM25Index,
vector_weight: f32,
keyword_weight: f32,
fusion: hybrid::Fusion,
k: usize,
exclude: Option<&[u8]>,
) -> Vec<(usize, f32)> {
self.ensure_hnsw_fresh();
let n = self.cache.len();
// Over-fetch so the merge sees a useful vector pool. `ef` is
// configurable, but the pool the fusion stage sees is not tied to it:
// a caller lowering `ef` for speed should not silently narrow what
// fusion has to work with.
let mut pool = (k * 8).max(64);
let mut allowed = n;
if let Some(ex) = exclude {
allowed = ex.iter().filter(|&&e| e == 0).count();
if allowed == 0 {
return Vec::new();
}
// Expect `pool` allowed hits if the filter is independent of the
// query's neighbourhood.
pool = pool.saturating_mul(n).div_ceil(allowed);
if allowed <= pool.saturating_mul(self.hnsw_m()) {
return self.exact_masked_search(query_embedding, query_text, bm25, fusion, k, ex);
}
}
match self.hnsw.as_ref() {
Some(index) if !index.is_empty() && index.dimension() == query_embedding.len() => {
// Over-fetch so the merge sees a useful vector pool; cosine
// distance from the index converts back to similarity (1 - d).
let pool = (k * 8).max(64);
let vec_scores: Vec<(usize, f32)> = index
.search(query_embedding, pool, pool)
let ef = self.hnsw_ef_search(k).max(pool);
let candidates = index.search(query_embedding, pool, ef);
// A quantised index returns approximate distances, and no
// amount of `ef` fixes that — the loss is in the distances,
// not the graph. Re-score the pool against the cache's exact
// embeddings, which cost nothing extra to keep: recall then
// matches an f32 index. See `BENCHMARKS.md`.
let exact = index.storage() == clawhdf5_ann::Storage::Int8;
let vec_scores: Vec<(usize, f32)> = candidates
.into_iter()
.map(|(id, dist)| (id, 1.0 - dist))
.filter(|(id, _)| exclude.is_none_or(|ex| ex[*id] == 0))
.map(|(id, dist)| {
let score = if exact {
crate::vector_search::cosine_similarity(
query_embedding,
&self.cache.embeddings[id],
)
} else {
1.0 - dist
};
(id, score)
})
.collect();
// Fusion normalises over every keyword match, so it needs all
// the scores — but not ranked.
let kw_scores = bm25.scores(query_text);
hybrid::merge_vector_keyword(
vec_scores,
kw_scores,
vector_weight,
keyword_weight,
let mut kw_scores = bm25.scores(query_text);
if let Some(ex) = exclude {
if vec_scores.len() < k.min(allowed) {
// The allowed records are not where the index looked.
return self.exact_masked_search(
query_embedding,
query_text,
bm25,
fusion,
k,
)
ex,
);
}
_ => hybrid::hybrid_search(
kw_scores.retain(|(id, _)| ex[*id] == 0);
}
hybrid::fuse(vec_scores, kw_scores, fusion, k)
}
_ => match exclude {
Some(ex) => {
self.exact_masked_search(query_embedding, query_text, bm25, fusion, k, ex)
}
None => hybrid::hybrid_search_fused(
query_embedding,
query_text,
&self.cache.embeddings,
&self.cache.chunks,
&self.cache.tombstones,
bm25,
vector_weight,
keyword_weight,
fusion,
k,
),
},
}
}
@@ -66,21 +201,52 @@ impl HDF5Memory {
query_embedding: &[f32],
query_text: &str,
bm25: &bm25::BM25Index,
vector_weight: f32,
keyword_weight: f32,
fusion: hybrid::Fusion,
k: usize,
exclude: Option<&[u8]>,
) -> Vec<(usize, f32)> {
hybrid::hybrid_search(
match exclude {
Some(ex) => self.exact_masked_search(query_embedding, query_text, bm25, fusion, k, ex),
None => hybrid::hybrid_search_fused(
query_embedding,
query_text,
&self.cache.embeddings,
&self.cache.chunks,
&self.cache.tombstones,
bm25,
vector_weight,
keyword_weight,
fusion,
k,
)
),
}
}
/// Exact hybrid search over the records `exclude` leaves (0 = allowed).
fn exact_masked_search(
&self,
query_embedding: &[f32],
query_text: &str,
bm25: &bm25::BM25Index,
fusion: hybrid::Fusion,
k: usize,
exclude: &[u8],
) -> Vec<(usize, f32)> {
let vec_scores =
hybrid::exact_vector_scores(query_embedding, &self.cache.embeddings, exclude);
let mut kw_scores = bm25.scores(query_text);
kw_scores.retain(|(id, _)| exclude.get(*id) == Some(&0));
hybrid::fuse(vec_scores, kw_scores, fusion, k)
}
/// The exclusion mask for a source-channel filter: 1 for a tombstoned
/// record or one from a channel not in `channels`.
fn source_mask(&self, channels: &[String]) -> Vec<u8> {
let allowed: HashSet<&str> = channels.iter().map(String::as_str).collect();
self.cache
.source_channels
.iter()
.zip(&self.cache.tombstones)
.map(|(ch, &t)| u8::from(t != 0 || !allowed.contains(ch.as_str())))
.collect()
}
/// Perform hybrid search combining cosine vector similarity and BM25 keyword search.
@@ -92,6 +258,61 @@ impl HDF5Memory {
keyword_weight: f32,
k: usize,
) -> Vec<SearchResult> {
self.hybrid_search_with(
query_embedding,
query_text,
hybrid::Fusion::Weighted {
vector: vector_weight,
keyword: keyword_weight,
},
k,
)
}
/// [`HDF5Memory::hybrid_search`] with the fusion method chosen explicitly.
///
/// [`hybrid::DEFAULT_FUSION`] is what the weighted form defaults to;
/// [`hybrid::Fusion::Rrf`] combines the two stages by rank instead of by
/// score.
pub fn hybrid_search_with(
&mut self,
query_embedding: &[f32],
query_text: &str,
fusion: hybrid::Fusion,
k: usize,
) -> Vec<SearchResult> {
self.search(
query_embedding,
query_text,
&SearchOptions::new(k).with_fusion(fusion),
)
}
/// Hybrid search with optional source filtering, re-ranking and
/// confidence rejection — see [`SearchOptions`].
///
/// Stages, in order: vector + keyword retrieval over the records the
/// source filter allows; fusion; scaling by Hebbian activation; re-ranking
/// (if on) of a `rerank_pool` of candidates; confidence rejection (if on);
/// the top `k`. The records returned with a positive score get their
/// Hebbian boost.
pub fn search(
&mut self,
query_embedding: &[f32],
query_text: &str,
options: &SearchOptions,
) -> Vec<SearchResult> {
let k = options.k;
let fetch = match options.rerank {
Some(_) if options.rerank_pool > 0 => options.rerank_pool.max(k),
Some(_) => k.saturating_mul(3).max(10),
None => k,
};
let exclude = options
.source_channels
.as_deref()
.map(|channels| self.source_mask(channels));
// The keyword index lives for the life of the store and is updated
// incrementally. Take it out for the duration of the call so the
// vector stage can borrow `self` mutably, then put it back.
@@ -101,10 +322,12 @@ impl HDF5Memory {
query_embedding,
query_text,
&bm25,
vector_weight,
keyword_weight,
k,
options.fusion,
fetch,
exclude.as_deref(),
);
self.bm25 = Some(bm25);
let mut results: Vec<SearchResult> = scored
.into_iter()
.map(|(idx, score)| {
@@ -128,6 +351,25 @@ impl HDF5Memory {
.then(a.index.cmp(&b.index))
});
if let Some(config) = &options.rerank {
results = Self::rerank_results(results, config, options.now);
}
if let Some(config) = &options.confidence {
let scored: Vec<ScoredResult> = results
.iter()
.map(|r| ScoredResult {
index: r.index,
score: r.score,
})
.collect();
let keep: HashSet<usize> = reject_low_confidence(&scored, config)
.into_iter()
.map(|r| r.index)
.collect();
results.retain(|r| keep.contains(&r.index));
}
results.truncate(k);
// Only reinforce records that actually matched. When fewer than `k`
// records are relevant, the rest of the list is zero-score filler;
// boosting it would teach the store that arbitrary records are
@@ -138,11 +380,45 @@ impl HDF5Memory {
.map(|r| r.index)
.collect();
self.apply_hebbian_boost(&hit_indices);
self.bm25 = Some(bm25);
results
}
/// Reorder by the re-ranker's combined score, which also becomes each
/// result's `score`.
fn rerank_results(
results: Vec<SearchResult>,
config: &ReRankConfig,
now: Option<f64>,
) -> Vec<SearchResult> {
let now = now.unwrap_or_else(|| {
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_secs_f64())
.unwrap_or(0.0)
});
let inputs: Vec<RerankInput> = results
.iter()
.map(|r| RerankInput {
index: r.index,
timestamp: r.timestamp,
source_channel: r.source_channel.clone(),
raw_activation: r.activation,
relevance: r.score,
})
.collect();
let mut by_index: std::collections::HashMap<usize, SearchResult> =
results.into_iter().map(|r| (r.index, r)).collect();
rerank(&inputs, config, now)
.into_iter()
.filter_map(|rr| {
let mut r = by_index.remove(&rr.index)?;
r.score = rr.combined_score;
Some(r)
})
.collect()
}
/// Reinforce the records a query returned. The new weights are persisted by
/// the next checkpoint (any write that flushes, `flush_wal`, or drop) — not
/// by rewriting the whole store inside the query, which is what made
crates/clawhdf5-agent/src/session.rs
+16 -1
View File
@@ -33,7 +33,7 @@ impl SessionCache {
self.entries.is_empty()
}
/// Add a new session with its summary.
/// Add a new session with its summary, timestamped now.
pub fn add(
&mut self,
id: &str,
@@ -47,6 +47,21 @@ impl SessionCache {
.unwrap_or_default()
.as_secs_f64()
* 1_000_000.0; // microseconds
self.add_at(id, start_idx, end_idx, channel, summary, ts);
}
/// Add a session with an explicit timestamp (Unix **microseconds**, the
/// unit [`SessionEntry::ts`] uses) — for importers carrying sessions over
/// from another store, whose original time should be kept.
pub fn add_at(
&mut self,
id: &str,
start_idx: usize,
end_idx: usize,
channel: &str,
summary: &str,
ts: f64,
) {
self.entries.push(SessionEntry {
id: id.to_string(),
start_idx: start_idx as u64,
crates/clawhdf5-agent/src/signing.rs
+419
View File
@@ -0,0 +1,419 @@
//! Ed25519-signed checkpoints.
//!
//! When a signing key is set ([`crate::HDF5Memory::set_signing_key`]), every
//! checkpoint writes a signed manifest of the store: a SHA-256 per memory
//! record rolled into a Merkle root, plus hashes of the store's settings, its
//! sessions and its knowledge graph. [`verify_store`] recomputes all of it from
//! the file and checks the signature against a public key the caller trusts,
//! so any change to the checkpointed file — a record's text or embedding, a
//! setting, a session, a graph edge, made through this crate or any other HDF5
//! tool — is detected, and the per-record hashes say which records changed.
//!
//! What it does not cover: saves still only in the WAL (made since the last
//! checkpoint). [`VerifyReport::wal_entries_unsigned`] counts them.
//!
//! The hashes cover exactly what the file persists, in the form the loader
//! returns it, so a store verifies after any number of reopen/checkpoint
//! cycles. Derived data (L2 norms, the vector index) is not covered; it is
//! recomputed from covered data.
use ed25519_dalek::{Signature, Signer, Verifier};
pub use ed25519_dalek::{SigningKey, VerifyingKey};
use sha2::{Digest, Sha256};
use crate::MemoryConfig;
use crate::cache::MemoryCache;
use crate::knowledge::KnowledgeCache;
use crate::session::SessionCache;
use crate::wal::WalMark;
/// Version of the manifest encoding; part of what is signed.
pub const MANIFEST_VERSION: i64 = 1;
type Hash = [u8; 32];
/// The hashes a signature covers.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct Manifest {
pub record_count: u64,
/// Merkle root over the per-record hashes.
pub records_root: Hash,
/// Settings persisted in `/meta`, plus the checkpoint's WAL mark.
pub settings: Hash,
pub sessions: Hash,
pub graph: Hash,
}
impl Manifest {
/// The exact bytes that are signed.
pub fn signed_bytes(&self) -> Vec<u8> {
let mut m = Vec::with_capacity(160);
m.extend_from_slice(b"clawhdf5-agent signed checkpoint\0");
m.extend_from_slice(&MANIFEST_VERSION.to_le_bytes());
m.extend_from_slice(&self.record_count.to_le_bytes());
m.extend_from_slice(&self.records_root);
m.extend_from_slice(&self.settings);
m.extend_from_slice(&self.sessions);
m.extend_from_slice(&self.graph);
m
}
}
/// A signature as stored in a checkpoint.
#[derive(Debug, Clone)]
pub struct StoredSignature {
pub manifest: Manifest,
pub record_hashes: Vec<Hash>,
pub public_key: [u8; 32],
pub signature: [u8; 64],
}
/// Build the manifest (and per-record hashes) for the state about to be
/// checkpointed, and sign it.
pub fn sign(
key: &SigningKey,
config: &MemoryConfig,
cache: &MemoryCache,
sessions: &SessionCache,
knowledge: &KnowledgeCache,
wal_applied: Option<WalMark>,
) -> StoredSignature {
let (manifest, record_hashes) = manifest(config, cache, sessions, knowledge, wal_applied);
let signature = key.sign(&manifest.signed_bytes()).to_bytes();
StoredSignature {
manifest,
record_hashes,
public_key: key.verifying_key().to_bytes(),
signature,
}
}
/// Compute the manifest of a store's state.
pub fn manifest(
config: &MemoryConfig,
cache: &MemoryCache,
sessions: &SessionCache,
knowledge: &KnowledgeCache,
wal_applied: Option<WalMark>,
) -> (Manifest, Vec<Hash>) {
let record_hashes: Vec<Hash> = (0..cache.len()).map(|i| record_hash(cache, i)).collect();
let manifest = Manifest {
record_count: cache.len() as u64,
records_root: merkle_root(&record_hashes),
settings: settings_hash(config, wal_applied),
sessions: sessions_hash(sessions),
graph: graph_hash(knowledge),
};
(manifest, record_hashes)
}
// ---------------------------------------------------------------------------
// Canonical encoding
// ---------------------------------------------------------------------------
/// A SHA-256 over length-prefixed fields, so no two different field lists
/// hash the same bytes.
struct Fields(Sha256);
impl Fields {
fn new(domain: &str) -> Self {
let mut h = Sha256::new();
h.update((domain.len() as u64).to_le_bytes());
h.update(domain.as_bytes());
Self(h)
}
fn bytes(&mut self, b: &[u8]) -> &mut Self {
self.0.update((b.len() as u64).to_le_bytes());
self.0.update(b);
self
}
/// Strings as the loader returns them: stored null-padded, so a trailing
/// NUL cannot survive a round trip and must not be part of the hash.
fn str(&mut self, s: &str) -> &mut Self {
self.bytes(s.trim_end_matches('\0').as_bytes())
}
fn u64(&mut self, v: u64) -> &mut Self {
self.0.update(v.to_le_bytes());
self
}
fn f64(&mut self, v: f64) -> &mut Self {
self.0.update(v.to_bits().to_le_bytes());
self
}
fn f32(&mut self, v: f32) -> &mut Self {
self.0.update(v.to_bits().to_le_bytes());
self
}
fn finish(self) -> Hash {
self.0.finalize().into()
}
}
/// Everything persisted about record `i`, including its position. The
/// embedding is hashed as the cache holds it — for a `float16` store that is
/// the half-rounded value the file holds.
fn record_hash(cache: &MemoryCache, i: usize) -> Hash {
let mut f = Fields::new("clawhdf5-agent/record");
f.u64(i as u64).str(&cache.chunks[i]);
let emb: Vec<u8> = cache.embeddings[i]
.iter()
.flat_map(|v| v.to_bits().to_le_bytes())
.collect();
f.bytes(&emb)
.str(&cache.source_channels[i])
.f64(cache.timestamps[i])
.str(&cache.session_ids[i])
.str(&cache.tags[i])
.u64(u64::from(cache.tombstones[i]))
.f32(cache.activation_weights[i]);
f.finish()
}
/// Binary Merkle tree: leaves are the record hashes; a parent hashes its two
/// children with a node prefix; an odd node is carried up unchanged.
fn merkle_root(leaves: &[Hash]) -> Hash {
if leaves.is_empty() {
return Fields::new("clawhdf5-agent/merkle-empty").finish();
}
let mut level: Vec<Hash> = leaves.to_vec();
while level.len() > 1 {
level = level
.chunks(2)
.map(|pair| match pair {
[l, r] => {
let mut h = Sha256::new();
h.update([1u8]);
h.update(l);
h.update(r);
h.finalize().into()
}
[only] => *only,
_ => unreachable!(),
})
.collect();
}
level[0]
}
fn settings_hash(c: &MemoryConfig, wal_applied: Option<WalMark>) -> Hash {
let mut f = Fields::new("clawhdf5-agent/settings");
f.str(crate::schema::SCHEMA_VERSION)
.str(&c.created_at)
.str(&c.agent_id)
.str(&c.embedder)
.u64(c.embedding_dim as u64)
.u64(c.chunk_size as u64)
.u64(c.overlap as u64)
.u64(u64::from(c.float16))
.u64(u64::from(c.compression))
.u64(u64::from(c.compression_level))
.f32(c.compact_threshold)
.f32(c.hebbian_boost)
.f32(c.decay_factor)
.u64(u64::from(c.wal_enabled))
.u64(c.wal_max_entries as u64)
.u64(u64::from(c.quantized_index))
.u64(c.hnsw_m as u64)
.u64(c.hnsw_ef_construction as u64)
.u64(c.hnsw_ef_search as u64);
// An empty mark is not written to the file, so it must hash as none.
match wal_applied.filter(|m| m.len > 0) {
Some(m) => f.u64(1).u64(m.len).u64(u64::from(m.crc)),
None => f.u64(0),
};
f.finish()
}
fn sessions_hash(s: &SessionCache) -> Hash {
let mut f = Fields::new("clawhdf5-agent/sessions");
f.u64(s.entries.len() as u64);
for (i, e) in s.entries.iter().enumerate() {
f.str(&e.id)
.u64(e.start_idx)
.u64(e.end_idx)
.str(&e.channel)
.f64(e.ts)
.str(s.summaries.get(i).map(String::as_str).unwrap_or(""));
}
f.finish()
}
fn graph_hash(k: &KnowledgeCache) -> Hash {
let mut f = Fields::new("clawhdf5-agent/graph");
f.u64(k.entities.len() as u64);
for e in &k.entities {
f.u64(e.id)
.str(&e.name)
.str(&e.entity_type)
.u64(e.embedding_idx as u64);
}
f.u64(k.relations.len() as u64);
for r in &k.relations {
f.u64(r.src)
.u64(r.tgt)
.str(&r.relation)
.f32(r.weight)
.f64(r.ts);
}
f.u64(k.alias_strings.len() as u64);
for (s, id) in k.alias_strings.iter().zip(&k.alias_entity_ids) {
f.str(s).u64(*id as u64);
}
f.finish()
}
// ---------------------------------------------------------------------------
// Verification
// ---------------------------------------------------------------------------
/// The outcome of [`verify_store`].
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct VerifyReport {
/// The checkpoint carries a signature.
pub signed: bool,
/// The signature was made by the key the caller trusts.
pub key_matches: bool,
/// The signature over the stored manifest is valid.
pub signature_valid: bool,
/// The file's current contents match the signed manifest.
pub records_match: bool,
pub settings_match: bool,
pub sessions_match: bool,
pub graph_match: bool,
/// Records whose contents differ from what was signed (by position),
/// when the stored per-record hashes are themselves authentic.
pub changed_records: Vec<usize>,
/// Records in the file versus in the signed manifest.
pub record_count: u64,
pub signed_record_count: u64,
/// The public key the checkpoint claims to be signed by.
pub public_key: Option<[u8; 32]>,
/// Saves in the WAL after the checkpoint: not covered by the signature.
pub wal_entries_unsigned: usize,
}
impl VerifyReport {
/// Signed by the trusted key, signature valid, and every part of the
/// file unchanged since it was signed.
pub fn is_valid(&self) -> bool {
self.signed
&& self.key_matches
&& self.signature_valid
&& self.records_match
&& self.settings_match
&& self.sessions_match
&& self.graph_match
}
}
/// Check a store file against the public key the caller trusts.
///
/// Reads the checkpoint (not the WAL), recomputes every hash from its
/// contents and checks the signature. Never writes.
pub fn verify_store(
path: &std::path::Path,
trusted: &VerifyingKey,
) -> Result<VerifyReport, crate::MemoryError> {
let file = clawhdf5::File::open(path)
.map_err(|e| crate::MemoryError::Hdf5(format!("cannot open {}: {e}", path.display())))?;
let (config, cache, sessions, knowledge) = crate::schema::validate_and_load(&file)?;
let checkpoint = crate::schema::read_checkpoint_meta(&file);
let stored = crate::schema::read_signature(&file)?;
let wal_entries_unsigned = count_wal_entries_after(path, checkpoint.wal_applied);
let (current, current_hashes) = manifest(
&config,
&cache,
&sessions,
&knowledge,
checkpoint.wal_applied,
);
let Some(stored) = stored else {
return Ok(VerifyReport {
signed: false,
key_matches: false,
signature_valid: false,
records_match: false,
settings_match: false,
sessions_match: false,
graph_match: false,
changed_records: Vec::new(),
record_count: current.record_count,
signed_record_count: 0,
public_key: None,
wal_entries_unsigned,
});
};
let key_matches = stored.public_key == trusted.to_bytes();
let signature_valid = trusted
.verify(
&stored.manifest.signed_bytes(),
&Signature::from_bytes(&stored.signature),
)
.is_ok();
// The stored per-record hashes can localise a change only if they are
// the ones that were signed.
let hashes_authentic = signature_valid
&& stored.record_hashes.len() as u64 == stored.manifest.record_count
&& merkle_root(&stored.record_hashes) == stored.manifest.records_root;
let changed_records = if hashes_authentic {
let n = current_hashes.len().max(stored.record_hashes.len());
(0..n)
.filter(|&i| current_hashes.get(i) != stored.record_hashes.get(i))
.collect()
} else {
Vec::new()
};
Ok(VerifyReport {
signed: true,
key_matches,
signature_valid,
records_match: signature_valid
&& current.record_count == stored.manifest.record_count
&& current.records_root == stored.manifest.records_root,
settings_match: signature_valid && current.settings == stored.manifest.settings,
sessions_match: signature_valid && current.sessions == stored.manifest.sessions,
graph_match: signature_valid && current.graph == stored.manifest.graph,
changed_records,
record_count: current.record_count,
signed_record_count: stored.manifest.record_count,
public_key: Some(stored.public_key),
wal_entries_unsigned,
})
}
fn count_wal_entries_after(store: &std::path::Path, mark: Option<WalMark>) -> usize {
let wal = store.with_extension("h5.wal");
if !wal.exists() {
return 0;
}
crate::wal::WalFile::read_entries_for_migration(&wal, mark)
.map(|e| e.len())
.unwrap_or(0)
}
/// A new random signing key from the operating system's RNG.
pub fn generate_key() -> SigningKey {
SigningKey::generate(&mut rand_core::OsRng)
}
/// Hex encoding for keys and signatures in attributes and the CLI.
pub fn to_hex(bytes: &[u8]) -> String {
bytes.iter().map(|b| format!("{b:02x}")).collect()
}
/// Parse hex into exactly `N` bytes.
pub fn from_hex<const N: usize>(s: &str) -> Option<[u8; N]> {
let s = s.trim();
if s.len() != 2 * N {
return None;
}
let mut out = [0u8; N];
for (i, byte) in out.iter_mut().enumerate() {
*byte = u8::from_str_radix(&s[2 * i..2 * i + 2], 16).ok()?;
}
Some(out)
}
crates/clawhdf5-agent/src/storage.rs
+22 -12
View File
@@ -36,7 +36,7 @@ pub fn write_to_disk_with_mark(
) -> Result<(), MemoryError> {
let meta = schema::CheckpointMeta {
wal_applied,
ann_generation: None,
..schema::CheckpointMeta::default()
};
write_to_disk_with_meta(path, config, cache, sessions, knowledge, &meta)
}
@@ -50,7 +50,21 @@ pub fn write_to_disk_with_meta(
knowledge: &KnowledgeCache,
checkpoint: &schema::CheckpointMeta,
) -> Result<(), MemoryError> {
let bytes = schema::build_hdf5_file_with_meta(config, cache, sessions, knowledge, checkpoint)?;
write_to_disk_signed(path, config, cache, sessions, knowledge, checkpoint, None)
}
/// [`write_to_disk_with_meta`] with a signed manifest of the contents.
pub fn write_to_disk_signed(
path: &Path,
config: &MemoryConfig,
cache: &MemoryCache,
sessions: &SessionCache,
knowledge: &KnowledgeCache,
checkpoint: &schema::CheckpointMeta,
signature: Option<&crate::signing::StoredSignature>,
) -> Result<(), MemoryError> {
let bytes =
schema::build_hdf5_file_signed(config, cache, sessions, knowledge, checkpoint, signature)?;
if bytes.is_empty() {
return Err(MemoryError::Hdf5("build_hdf5_file produced 0 bytes".into()));
@@ -113,13 +127,11 @@ pub type StoreState = (MemoryConfig, MemoryCache, SessionCache, KnowledgeCache);
/// [`read_from_disk`], plus the checkpoint's [`WalMark`] (if any) so the
/// caller can skip WAL entries this file already contains.
pub fn read_from_disk_with_mark(path: &Path) -> Result<(StoreState, Option<WalMark>), MemoryError> {
let mmap = clawhdf5_io::MmapReader::open(path).map_err(MemoryError::Io)?;
// Advise the OS we'll need the whole file for parsing
mmap.advise_willneed(0, mmap.len());
// Parse the HDF5 file from the mmap'd bytes
let file = clawhdf5::File::from_bytes(mmap.as_bytes().to_vec())
// `File::open` memory-maps the file itself (the facade's `mmap` feature is
// on by default). Mapping it here and handing over `as_bytes().to_vec()`
// did the same work and then copied the whole store — a second full copy
// of the file, live for the whole parse, on top of the mapping.
let file = clawhdf5::File::open(path)
.map_err(|e| MemoryError::Hdf5(format!("cannot open {}: {e}", path.display())))?;
let (mut config, cache, sessions, knowledge) = schema::validate_and_load(&file)?;
@@ -133,9 +145,7 @@ pub fn read_from_disk_with_mark(path: &Path) -> Result<(StoreState, Option<WalMa
pub fn read_from_disk_with_meta(
path: &Path,
) -> Result<(StoreState, schema::CheckpointMeta), MemoryError> {
let mmap = clawhdf5_io::MmapReader::open(path).map_err(MemoryError::Io)?;
mmap.advise_willneed(0, mmap.len());
let file = clawhdf5::File::from_bytes(mmap.as_bytes().to_vec())
let file = clawhdf5::File::open(path)
.map_err(|e| MemoryError::Hdf5(format!("cannot open {}: {e}", path.display())))?;
let (mut config, cache, sessions, knowledge) = schema::validate_and_load(&file)?;
config.path = path.to_path_buf();
crates/clawhdf5-agent/src/vector_search.rs
+75 -30
View File
@@ -4,6 +4,44 @@
//! `clawhdf5_accel`, with optional float16 support via the `half` crate.
//! Supports pre-computed norms for eliminating redundant norm computations.
/// A corpus of equal-length embeddings addressable by index.
///
/// Lets the batch kernels read either the cache's flat `[N x dim]` buffer or a
/// plain `Vec<Vec<f32>>` without either side owning a second copy.
pub trait VectorSet {
/// Number of embeddings.
fn count(&self) -> usize;
/// Embedding `i`; callers only index below [`VectorSet::count`].
fn row(&self, i: usize) -> &[f32];
}
impl VectorSet for [Vec<f32>] {
fn count(&self) -> usize {
self.len()
}
fn row(&self, i: usize) -> &[f32] {
&self[i]
}
}
impl VectorSet for Vec<Vec<f32>> {
fn count(&self) -> usize {
self.len()
}
fn row(&self, i: usize) -> &[f32] {
&self[i]
}
}
impl VectorSet for crate::cache::Embeddings {
fn count(&self) -> usize {
self.len()
}
fn row(&self, i: usize) -> &[f32] {
&self[i]
}
}
/// Compute cosine similarity between two f32 slices.
///
/// Returns 0.0 if either vector has zero magnitude.
@@ -22,7 +60,7 @@ pub fn cosine_similarity(a: &[f32], b: &[f32]) -> f32 {
/// Returns `(index, score)` pairs sorted by score descending.
pub fn cosine_similarity_batch(
query: &[f32],
vectors: &[Vec<f32>],
vectors: &(impl VectorSet + ?Sized),
tombstones: &[u8],
) -> Vec<(usize, f32)> {
let query_norm = clawhdf5_accel::vector_norm(query);
@@ -30,7 +68,7 @@ pub fn cosine_similarity_batch(
return Vec::new();
}
let n = vectors.len();
let n = vectors.count();
let mut results: Vec<(usize, f32)> = Vec::with_capacity(n);
// Process 4 vectors at a time where possible
@@ -42,8 +80,9 @@ pub fn cosine_similarity_batch(
if i < tombstones.len() && tombstones[i] != 0 {
continue;
}
let vec_norm = clawhdf5_accel::vector_norm(&vectors[i]);
let score = crate::cosine_similarity_prenorm(query, query_norm, &vectors[i], vec_norm);
let vec_norm = clawhdf5_accel::vector_norm(vectors.row(i));
let score =
crate::cosine_similarity_prenorm(query, query_norm, vectors.row(i), vec_norm);
results.push((i, score));
}
}
@@ -53,8 +92,8 @@ pub fn cosine_similarity_batch(
if i < tombstones.len() && tombstones[i] != 0 {
continue;
}
let vec_norm = clawhdf5_accel::vector_norm(&vectors[i]);
let score = crate::cosine_similarity_prenorm(query, query_norm, &vectors[i], vec_norm);
let vec_norm = clawhdf5_accel::vector_norm(vectors.row(i));
let score = crate::cosine_similarity_prenorm(query, query_norm, vectors.row(i), vec_norm);
results.push((i, score));
}
@@ -68,7 +107,7 @@ pub fn cosine_similarity_batch(
/// collections. Uses `score = dot(query, vec) / (query_norm * stored_norm)`.
pub fn cosine_similarity_batch_prenorm(
query: &[f32],
vectors: &[Vec<f32>],
vectors: &(impl VectorSet + ?Sized),
norms: &[f32],
tombstones: &[u8],
) -> Vec<(usize, f32)> {
@@ -77,7 +116,7 @@ pub fn cosine_similarity_batch_prenorm(
return Vec::new();
}
let n = vectors.len();
let n = vectors.count();
let mut results: Vec<(usize, f32)> = Vec::with_capacity(n);
for i in 0..n {
@@ -85,7 +124,7 @@ pub fn cosine_similarity_batch_prenorm(
continue;
}
let vec_norm = norms[i];
let score = crate::cosine_similarity_prenorm(query, query_norm, &vectors[i], vec_norm);
let score = crate::cosine_similarity_prenorm(query, query_norm, vectors.row(i), vec_norm);
results.push((i, score));
}
@@ -162,7 +201,7 @@ pub fn cosine_similarity_f16(
#[cfg(feature = "parallel")]
pub fn parallel_cosine_batch(
query: &[f32],
vectors: &[Vec<f32>],
vectors: &(impl VectorSet + Sync + ?Sized),
tombstones: &[u8],
k: usize,
) -> Vec<(usize, f32)> {
@@ -174,24 +213,27 @@ pub fn parallel_cosine_batch(
}
let num_cores = rayon::current_num_threads().max(1);
let chunk_size = vectors.len().div_ceil(num_cores);
let chunk_size = vectors.count().div_ceil(num_cores);
if chunk_size == 0 {
return Vec::new();
}
let mut all_results: Vec<(usize, f32)> = vectors
.par_chunks(chunk_size)
.enumerate()
.flat_map(|(chunk_idx, chunk)| {
// Chunk over index ranges: the corpus may be one flat buffer rather than
// a slice of rows, so there is nothing to `par_chunks` over.
let n = vectors.count();
let mut all_results: Vec<(usize, f32)> = (0..n.div_ceil(chunk_size))
.into_par_iter()
.flat_map(|chunk_idx| {
let base = chunk_idx * chunk_size;
let mut local: Vec<(usize, f32)> = Vec::with_capacity(chunk.len());
for (j, vec) in chunk.iter().enumerate() {
let i = base + j;
let end = (base + chunk_size).min(n);
let mut local: Vec<(usize, f32)> = Vec::with_capacity(end - base);
for i in base..end {
if i < tombstones.len() && tombstones[i] != 0 {
continue;
}
let vec_norm = clawhdf5_accel::vector_norm(vec);
let score = crate::cosine_similarity_prenorm(query, query_norm, vec, vec_norm);
let vec_norm = clawhdf5_accel::vector_norm(vectors.row(i));
let score =
crate::cosine_similarity_prenorm(query, query_norm, vectors.row(i), vec_norm);
local.push((i, score));
}
local.sort_by(|a, b| b.1.partial_cmp(&a.1).unwrap_or(std::cmp::Ordering::Equal));
@@ -209,7 +251,7 @@ pub fn parallel_cosine_batch(
#[cfg(feature = "parallel")]
pub fn parallel_cosine_batch_prenorm(
query: &[f32],
vectors: &[Vec<f32>],
vectors: &(impl VectorSet + Sync + ?Sized),
norms: &[f32],
tombstones: &[u8],
k: usize,
@@ -222,23 +264,26 @@ pub fn parallel_cosine_batch_prenorm(
}
let num_cores = rayon::current_num_threads().max(1);
let chunk_size = vectors.len().div_ceil(num_cores);
let chunk_size = vectors.count().div_ceil(num_cores);
if chunk_size == 0 {
return Vec::new();
}
let mut all_results: Vec<(usize, f32)> = vectors
.par_chunks(chunk_size)
.enumerate()
.flat_map(|(chunk_idx, chunk)| {
// Chunk over index ranges: the corpus may be one flat buffer rather than
// a slice of rows, so there is nothing to `par_chunks` over.
let n = vectors.count();
let mut all_results: Vec<(usize, f32)> = (0..n.div_ceil(chunk_size))
.into_par_iter()
.flat_map(|chunk_idx| {
let base = chunk_idx * chunk_size;
let mut local: Vec<(usize, f32)> = Vec::with_capacity(chunk.len());
for (j, vec) in chunk.iter().enumerate() {
let i = base + j;
let end = (base + chunk_size).min(n);
let mut local: Vec<(usize, f32)> = Vec::with_capacity(end - base);
for i in base..end {
if i < tombstones.len() && tombstones[i] != 0 {
continue;
}
let score = crate::cosine_similarity_prenorm(query, query_norm, vec, norms[i]);
let score =
crate::cosine_similarity_prenorm(query, query_norm, vectors.row(i), norms[i]);
local.push((i, score));
}
local.sort_by(|a, b| b.1.partial_cmp(&a.1).unwrap_or(std::cmp::Ordering::Equal));
crates/clawhdf5-agent/tests/fixtures/store_v2_5_0.h5
Vendored
BIN
View File
Binary file not shown.
crates/clawhdf5-agent/tests/float16_store.rs
+300
View File
@@ -0,0 +1,300 @@
//! `MemoryConfig::float16`: embeddings stored as IEEE half precision.
//!
//! The setting used to be recorded in `/meta` and otherwise ignored — the
//! embeddings dataset was always `f32`. These tests pin what it now does: the
//! dataset is `float16`, the in-memory cache holds exactly the values the file
//! holds (so search results survive a reopen bit for bit), and a value half
//! precision cannot represent is refused rather than stored as infinity.
use std::path::{Path, PathBuf};
use clawhdf5_agent::{AgentMemory, HDF5Memory, MemoryConfig, MemoryEntry, MemoryError};
use clawhdf5_format::float16::round_to_f16;
use tempfile::TempDir;
const DIM: usize = 64;
/// Deterministic, embedding-like unit vectors.
fn embedding(seed: u64) -> Vec<f32> {
let mut x = seed.wrapping_mul(0x9E37_79B9_7F4A_7C15) | 1;
let v: Vec<f32> = (0..DIM)
.map(|_| {
x ^= x << 13;
x ^= x >> 7;
x ^= x << 17;
(x >> 40) as f32 / (1u64 << 24) as f32 - 0.5
})
.collect();
let norm = v.iter().map(|a| a * a).sum::<f32>().sqrt();
v.iter().map(|a| a / norm).collect()
}
fn entry(i: u64) -> MemoryEntry {
MemoryEntry {
chunk: format!("memory number {i} about topic {}", i % 7),
embedding: embedding(i),
source_channel: "test".into(),
timestamp: i as f64,
session_id: "s".into(),
tags: format!("t{i}"),
}
}
fn config(dir: &TempDir, name: &str, float16: bool) -> MemoryConfig {
let mut c = MemoryConfig::new(dir.path().join(name), "agent", DIM);
c.float16 = float16;
c
}
fn embeddings_dtype_and_values(path: &Path) -> (String, Vec<f32>) {
let file = clawhdf5::File::open(path).unwrap();
let ds = file.dataset("memory/embeddings").unwrap();
(format!("{:?}", ds.dtype().unwrap()), ds.read_f32().unwrap())
}
fn search_bits(m: &mut HDF5Memory, q: u64) -> Vec<(usize, u32)> {
m.hybrid_search(&embedding(q), "memory topic 3", 0.4, 0.6, 10)
.iter()
.map(|r| (r.index, r.score.to_bits()))
.collect()
}
#[test]
fn float16_store_writes_half_precision_and_reopens_identically() {
let dir = TempDir::new().unwrap();
// Two identical stores. Search is not read-only (it boosts the Hebbian
// activation of what it returns, and checkpoints persist that), so each
// is queried exactly once: one live, one after a checkpoint and reopen.
let live_cfg = config(&dir, "live.h5", true);
let cfg = config(&dir, "f16.h5", true);
let path: PathBuf = cfg.path.clone();
let mut live = HDF5Memory::create(live_cfg).unwrap();
live.save_batch((0..200).map(entry).collect()).unwrap();
let mut m = HDF5Memory::create(cfg).unwrap();
m.save_batch((0..200).map(entry).collect()).unwrap();
drop(m);
// On disk: a genuine float16 dataset holding the rounded inputs.
let (dtype, values) = embeddings_dtype_and_values(&path);
assert_eq!(dtype, "Other(\"float16\")");
let expected: Vec<u32> = (0..200)
.flat_map(|i| embedding(i).into_iter().map(|v| round_to_f16(v).to_bits()))
.collect();
let got: Vec<u32> = values.iter().map(|v| v.to_bits()).collect();
assert_eq!(got, expected);
// Reopened, the store answers exactly as the live one does: the cache
// held the half-rounded values before the checkpoint.
let mut reopened = HDF5Memory::open(&path).unwrap();
for q in 0..5 {
assert_eq!(
search_bits(&mut live, 1000 + q),
search_bits(&mut reopened, 1000 + q),
"query {q}"
);
}
}
#[test]
fn float16_halves_the_embeddings_on_disk() {
let dir = TempDir::new().unwrap();
let mut sizes = Vec::new();
for float16 in [false, true] {
let cfg = config(&dir, &format!("s{float16}.h5"), float16);
let path = cfg.path.clone();
let mut m = HDF5Memory::create(cfg).unwrap();
m.save_batch((0..2000).map(entry).collect()).unwrap();
drop(m);
sizes.push(std::fs::metadata(&path).unwrap().len());
}
let embedding_bytes_f32 = (2000 * DIM * 4) as u64;
let saved = sizes[0] - sizes[1];
// Half of the f32 embeddings, give or take metadata and alignment.
assert!(
saved.abs_diff(embedding_bytes_f32 / 2) < 16 * 1024,
"f32 {} B, f16 {} B, saved {saved} B, expected ~{} B",
sizes[0],
sizes[1],
embedding_bytes_f32 / 2
);
}
#[test]
fn f32_store_is_unchanged() {
let dir = TempDir::new().unwrap();
let cfg = config(&dir, "f32.h5", false);
let path = cfg.path.clone();
let mut m = HDF5Memory::create(cfg).unwrap();
m.save_batch((0..50).map(entry).collect()).unwrap();
drop(m);
let (dtype, values) = embeddings_dtype_and_values(&path);
assert_eq!(dtype, "F32");
let expected: Vec<f32> = (0..50).flat_map(embedding).collect();
assert_eq!(values, expected);
}
#[test]
fn out_of_range_values_are_refused_not_stored_as_infinity() {
let dir = TempDir::new().unwrap();
let mut cfg = config(&dir, "range.h5", true);
cfg.wal_enabled = true;
let path = cfg.path.clone();
let mut m = HDF5Memory::create(cfg).unwrap();
m.save(entry(1)).unwrap();
let mut bad = entry(2);
bad.embedding[5] = 70_000.0;
match m.save(bad.clone()) {
Err(MemoryError::InvalidEntry(msg)) => assert!(msg.contains("embedding[5]"), "{msg}"),
other => panic!("expected InvalidEntry, got {other:?}"),
}
assert!(matches!(
m.save_or_update(bad.clone()),
Err(MemoryError::InvalidEntry(_))
));
// A batch is all or nothing.
assert!(matches!(
m.save_batch(vec![entry(3), bad.clone(), entry(4)]),
Err(MemoryError::InvalidEntry(_))
));
assert_eq!(m.count(), 1);
// The largest finite half, and values that round down to it, are fine.
let mut edge = entry(5);
edge.embedding[0] = 65504.0;
edge.embedding[1] = -65519.0;
m.save(edge).unwrap();
assert_eq!(m.count(), 2);
drop(m);
// Nothing rejected reached the WAL or the file.
let m = HDF5Memory::open(&path).unwrap();
assert_eq!(m.count(), 2);
// An f32 store takes the same value as it always did.
let mut m32 = HDF5Memory::create(config(&dir, "range32.h5", false)).unwrap();
m32.save(bad).unwrap();
}
#[test]
fn wal_replay_rounds_like_a_live_save() {
let dir = TempDir::new().unwrap();
let mut cfg = config(&dir, "wal.h5", true);
cfg.wal_enabled = true;
cfg.wal_max_entries = 10_000; // keep everything in the WAL
let path = cfg.path.clone();
let mut m = HDF5Memory::create(cfg).unwrap();
for i in 0..30 {
m.save(entry(i)).unwrap();
}
let live = search_bits(&mut m, 77);
// Crash image: the .h5 is still the empty checkpoint; everything is in
// the WAL, which holds the caller's f32 values.
let crash = TempDir::new().unwrap();
let image = crash.path().join("image.h5");
std::fs::copy(&path, &image).unwrap();
std::fs::copy(
path.with_extension("h5.wal"),
image.with_extension("h5.wal"),
)
.unwrap();
drop(m);
let mut recovered = HDF5Memory::open(&image).unwrap();
assert_eq!(recovered.count(), 30);
assert_eq!(search_bits(&mut recovered, 77), live);
}
#[test]
fn new_stores_default_to_float16() {
let dir = TempDir::new().unwrap();
let path = dir.path().join("default.h5");
let mut m = HDF5Memory::create(MemoryConfig::new(path.clone(), "agent", DIM)).unwrap();
assert!(m.config().float16);
m.save_batch((0..10).map(entry).collect()).unwrap();
drop(m);
assert_eq!(embeddings_dtype_and_values(&path).0, "Other(\"float16\")");
assert!(HDF5Memory::open(&path).unwrap().config().float16);
}
#[test]
fn an_existing_f32_store_stays_f32() {
// Written by the v2.5.0 CLI, with `float16 = 0` in /meta (every agent
// store has recorded it). Flipping the default for new stores must not
// reach back and round an existing store's embeddings.
let dir = TempDir::new().unwrap();
let path = dir.path().join("legacy.h5");
std::fs::copy(
concat!(
env!("CARGO_MANIFEST_DIR"),
"/tests/fixtures/store_v2_5_0.h5"
),
&path,
)
.unwrap();
let before = embeddings_dtype_and_values(&path);
assert_eq!(before.0, "F32");
let mut m = HDF5Memory::open(&path).unwrap();
assert!(!m.config().float16, "an old store must reopen as f32");
let dim = m.config().embedding_dim;
let odd: Vec<f32> = (0..dim).map(|i| 0.1 + i as f32 * 1e-4).collect();
m.save_batch(vec![MemoryEntry {
chunk: "added after the upgrade".into(),
embedding: odd.clone(),
source_channel: "test".into(),
timestamp: 1.0,
session_id: "s".into(),
tags: String::new(),
}])
.unwrap();
drop(m);
// Checkpointed: still f32, the old rows untouched and the new one exact.
let (dtype, values) = embeddings_dtype_and_values(&path);
assert_eq!(dtype, "F32");
assert_eq!(&values[..before.1.len()], before.1.as_slice());
assert_eq!(&values[before.1.len()..], odd.as_slice());
}
/// `Group::attrs` leaves out an attribute it cannot decode. A store whose
/// `float16` setting is unreadable must not open as `float16 = false` (or with
/// any other default in place of a setting it has): it is an error.
#[test]
fn unreadable_meta_attribute_fails_open_instead_of_defaulting() {
let dir = TempDir::new().unwrap();
let path = dir.path().join("store.h5");
{
let mut m = HDF5Memory::create(config(&dir, "store.h5", true)).unwrap();
m.save(entry(1)).unwrap();
m.flush_wal().unwrap();
}
assert!(HDF5Memory::open_read_only(&path).is_ok());
// Give the `float16` attribute message an unknown version (the name is
// at +8 in a version-1 message and +9 in a version-3 one).
let mut bytes = std::fs::read(&path).unwrap();
let name = b"float16\0";
let mut hit = false;
let positions: Vec<usize> = (9..bytes.len() - name.len())
.filter(|&p| &bytes[p..p + name.len()] == name)
.collect();
for pos in positions {
for (back, version) in [(8, 1u8), (9, 3u8)] {
if bytes[pos - back] == version {
bytes[pos - back] = 0x7f;
hit = true;
}
}
}
assert!(hit, "float16 attribute message not found");
std::fs::write(&path, &bytes).unwrap();
match HDF5Memory::open_read_only(&path) {
Err(MemoryError::Schema(msg)) => assert!(msg.contains("/meta"), "{msg}"),
Err(e) => panic!("unexpected error: {e}"),
Ok(_) => panic!("store opened with an unreadable float16 setting"),
}
}
crates/clawhdf5-agent/tests/h5py_interop.rs
+155
View File
@@ -0,0 +1,155 @@
//! An agent store is a standard HDF5 file: h5py can open it and read every
//! dataset.
//!
//! It could not: the float datatype's sign-bit position was hard-coded for
//! f64, so every f32 dataset (embeddings, norms, activation weights) made
//! libhdf5 refuse the file with "sign bit position out of bounds".
use std::process::Command;
use clawhdf5_agent::{AgentMemory, HDF5Memory, MemoryConfig, MemoryEntry};
fn python() -> String {
std::env::var("CLAWHDF5_PYTHON").unwrap_or_else(|_| "python3".to_string())
}
fn h5py_available() -> bool {
Command::new(python())
.args(["-c", "import h5py"])
.output()
.map(|o| o.status.success())
.unwrap_or(false)
}
#[test]
fn h5py_reads_every_dataset_of_an_agent_store() {
if !h5py_available() {
assert!(
std::env::var("CLAWHDF5_REQUIRE_INTEROP").as_deref() != Ok("1"),
"CLAWHDF5_REQUIRE_INTEROP=1 but python3 with h5py is not available"
);
eprintln!("SKIP: python3 with h5py not available");
return;
}
let dir = tempfile::tempdir().unwrap();
for float16 in [false, true] {
let path = dir.path().join(format!("store_{float16}.h5"));
let mut cfg = MemoryConfig::new(path.clone(), "agent", 8);
cfg.float16 = float16;
let mut m = HDF5Memory::create(cfg).unwrap();
// save_batch checkpoints, so the records are in the .h5, not the WAL.
m.save_batch(
(0..20)
.map(|i| MemoryEntry {
chunk: format!("memory {i}"),
embedding: (0..8).map(|j| ((i * 8 + j) as f32).sin()).collect(),
source_channel: "test".into(),
timestamp: i as f64,
session_id: "s".into(),
tags: String::new(),
})
.collect(),
)
.unwrap();
drop(m);
// Exact expected values, as bits: numpy's sin need not match Rust's
// to the last place.
let bits = (0..160)
.map(|k| (k as f32).sin().to_bits().to_string())
.collect::<Vec<_>>()
.join(",");
let script = format!(
r#"
import h5py, numpy as np
want = np.float16 if {py_bool} else np.float32
with h5py.File("{path}", "r") as f:
names = []
f.visititems(lambda n, o: names.append(n) if isinstance(o, h5py.Dataset) else None)
for n in names:
f[n][()] # every dataset must decode
e = f["memory/embeddings"]
assert e.dtype == want, e.dtype
assert e.shape == (20, 8), e.shape
ref = np.array([{bits}], dtype=np.uint32).view(np.float32).astype(want).reshape(20, 8)
assert (e[()] == ref).all()
assert f["memory/norms"].dtype == np.float32
print(len(names))
"#,
py_bool = if float16 { "True" } else { "False" },
path = path.display()
);
let out = Command::new(python())
.args(["-c", &script])
.output()
.unwrap();
assert!(
out.status.success(),
"float16={float16}: {}",
String::from_utf8_lossy(&out.stderr)
);
let n: usize = String::from_utf8_lossy(&out.stdout).trim().parse().unwrap();
assert!(n >= 10, "only {n} datasets");
}
}
#[test]
fn an_edit_made_with_h5py_breaks_the_signature_and_names_the_record() {
if !h5py_available() {
assert!(
std::env::var("CLAWHDF5_REQUIRE_INTEROP").as_deref() != Ok("1"),
"CLAWHDF5_REQUIRE_INTEROP=1 but python3 with h5py is not available"
);
eprintln!("SKIP: python3 with h5py not available");
return;
}
use clawhdf5_agent::signing::SigningKey;
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("signed.h5");
let key = SigningKey::from_bytes(&[42; 32]);
let mut m = HDF5Memory::create(MemoryConfig::new(path.clone(), "agent", 8)).unwrap();
m.set_signing_key(key.clone());
m.save_batch(
(0..10)
.map(|i| MemoryEntry {
chunk: format!("memory {i}"),
embedding: (0..8).map(|j| ((i * 8 + j) as f32).cos()).collect(),
source_channel: "test".into(),
timestamp: i as f64,
session_id: "s".into(),
tags: String::new(),
})
.collect(),
)
.unwrap();
drop(m);
assert!(
HDF5Memory::verify(&path, &key.verifying_key())
.unwrap()
.is_valid()
);
// Someone edits one timestamp in place with h5py.
let script = format!(
r#"
import h5py
with h5py.File("{}", "r+") as f:
ts = f["memory/timestamps"]
ts[3] = 12345.0
"#,
path.display()
);
let out = Command::new(python())
.args(["-c", &script])
.output()
.unwrap();
assert!(
out.status.success(),
"{}",
String::from_utf8_lossy(&out.stderr)
);
let r = HDF5Memory::verify(&path, &key.verifying_key()).unwrap();
assert!(r.signature_valid && !r.is_valid(), "{r:?}");
assert_eq!(r.changed_records, vec![3]);
}
crates/clawhdf5-agent/tests/hnsw_integration.rs
+179
View File
@@ -165,3 +165,182 @@ fn save_batch_then_search_is_consistent() {
);
}
}
#[test]
fn quantized_index_matches_the_f32_index_after_re_scoring() {
// A quantised index holds approximate vectors, but the store still has the
// exact ones, so the query path re-scores the candidate pool before
// fusion. The results a caller sees should therefore be the same.
let dim = 64;
let n = 400;
let mut seed = 0x5EED_1234_5678_9ABC;
let vectors: Vec<Vec<f32>> = (0..n).map(|_| make_vector(&mut seed, dim)).collect();
let queries: Vec<Vec<f32>> = (0..20).map(|_| make_vector(&mut seed, dim)).collect();
let build = |dir: &TempDir, quantized: bool| {
let mut config = MemoryConfig::new(dir.path().join("mem.h5"), "agent", dim);
config.quantized_index = quantized;
let mut mem = HDF5Memory::create(config).unwrap();
for (i, v) in vectors.iter().enumerate() {
mem.save(entry(&format!("chunk {i}"), v.clone(), &format!("k{i}")))
.unwrap();
}
mem
};
let exact_dir = TempDir::new().unwrap();
let quant_dir = TempDir::new().unwrap();
let mut exact = build(&exact_dir, false);
let mut quantized = build(&quant_dir, true);
let k = 10;
let mut agree = 0;
for q in &queries {
let want: Vec<usize> = exact
.hybrid_search(q, "", 1.0, 0.0, k)
.iter()
.map(|r| r.index)
.collect();
agree += quantized
.hybrid_search(q, "", 1.0, 0.0, k)
.iter()
.filter(|r| want.contains(&r.index))
.count();
}
let overlap = agree as f64 / (k * queries.len()) as f64;
assert!(
overlap >= 0.95,
"quantised store should match the f32 one: {overlap}"
);
}
#[test]
fn quantized_index_setting_survives_a_reopen() {
let dir = TempDir::new().unwrap();
let path = dir.path().join("mem.h5");
let mut config = MemoryConfig::new(path.clone(), "agent", 8);
config.quantized_index = true;
let mut mem = HDF5Memory::create(config).unwrap();
let mut seed = 7;
for i in 0..30 {
mem.save(entry(&format!("c{i}"), make_vector(&mut seed, 8), "t"))
.unwrap();
}
mem.flush_wal().unwrap();
drop(mem);
// Reopening must not silently quadruple the index's memory, so the flag
// is part of the stored config rather than a per-session choice.
let reopened = HDF5Memory::open(&path).unwrap();
assert!(reopened.config().quantized_index);
}
#[test]
fn hnsw_parameters_are_configurable_and_persisted() {
// The graph degree and both candidate-list sizes used to be constants, so
// a deployment could not trade recall against memory or speed at all.
let dir = TempDir::new().unwrap();
let path = dir.path().join("mem.h5");
let mut config = MemoryConfig::new(path.clone(), "agent", 16);
config.hnsw_m = 8;
config.hnsw_ef_construction = 32;
config.hnsw_ef_search = 128;
let mut mem = HDF5Memory::create(config).unwrap();
let mut seed = 99;
let vectors: Vec<Vec<f32>> = (0..300).map(|_| make_vector(&mut seed, 16)).collect();
for (i, v) in vectors.iter().enumerate() {
mem.save(entry(&format!("c{i}"), v.clone(), "t")).unwrap();
}
// Still correct with a smaller graph: an exact match must rank first.
let top = mem.hybrid_search(&vectors[42], "", 1.0, 0.0, 1);
assert_eq!(top[0].index, 42);
mem.flush_wal().unwrap();
drop(mem);
let reopened = HDF5Memory::open(&path).unwrap();
assert_eq!(reopened.config().hnsw_m, 8);
assert_eq!(reopened.config().hnsw_ef_construction, 32);
assert_eq!(reopened.config().hnsw_ef_search, 128);
}
#[test]
fn degenerate_hnsw_parameters_do_not_panic() {
// `clawhdf5-ann` asserts m >= 2, so a zero from a config file — or from a
// caller who assumed 0 meant "default" — would abort the process inside
// the index builder. The store clamps instead.
let dir = TempDir::new().unwrap();
let mut config = MemoryConfig::new(dir.path().join("mem.h5"), "agent", 8);
config.hnsw_m = 0;
config.hnsw_ef_construction = 0;
config.hnsw_ef_search = 1;
let mut mem = HDF5Memory::create(config).unwrap();
let mut seed = 5;
let vectors: Vec<Vec<f32>> = (0..50).map(|_| make_vector(&mut seed, 8)).collect();
for (i, v) in vectors.iter().enumerate() {
mem.save(entry(&format!("c{i}"), v.clone(), "t")).unwrap();
}
let results = mem.hybrid_search(&vectors[7], "", 1.0, 0.0, 5);
assert_eq!(results[0].index, 7, "exact match should still rank first");
}
#[test]
fn new_stores_default_to_the_quantized_index() {
// int8 is the default because it is smaller and, with an exact re-score,
// faster at equal recall on every platform measured (see BENCHMARKS.md).
let dir = TempDir::new().unwrap();
let config = MemoryConfig::new(dir.path().join("mem.h5"), "agent", 8);
assert!(config.quantized_index);
let path = config.path.clone();
let mut mem = HDF5Memory::create(config).unwrap();
let mut seed = 3;
let vectors: Vec<Vec<f32>> = (0..40).map(|_| make_vector(&mut seed, 8)).collect();
for (i, v) in vectors.iter().enumerate() {
mem.save(entry(&format!("c{i}"), v.clone(), "t")).unwrap();
}
assert_eq!(
mem.hybrid_search(&vectors[11], "", 1.0, 0.0, 1)[0].index,
11
);
mem.flush_wal().unwrap();
drop(mem);
assert!(HDF5Memory::open(&path).unwrap().config().quantized_index);
}
#[test]
fn a_store_written_before_the_setting_existed_stays_f32() {
// `store_v2_5_0.h5` was written by the v2.5.0 CLI, before
// `quantized_index` or the HNSW parameters were persisted, so it carries
// none of them. Flipping the default for new stores must not reach back
// and change how an existing store's index is held.
let dir = TempDir::new().unwrap();
let path = dir.path().join("legacy.h5");
std::fs::copy(
concat!(
env!("CARGO_MANIFEST_DIR"),
"/tests/fixtures/store_v2_5_0.h5"
),
&path,
)
.unwrap();
let bytes = std::fs::read(&path).unwrap();
assert!(
!bytes.windows(15).any(|w| w == b"quantized_index"),
"the fixture must predate the setting, or it tests nothing"
);
let mut mem = HDF5Memory::open(&path).unwrap();
assert!(
!mem.config().quantized_index,
"an old store must reopen with an f32 index"
);
assert_eq!(mem.config().hnsw_m, 16);
assert_eq!(mem.config().hnsw_ef_construction, 64);
assert_eq!(mem.count(), 6);
// And it still searches: entry 3's own embedding finds it first.
let hit = mem.hybrid_search(&[3.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], "", 1.0, 0.0, 1);
assert_eq!(hit[0].index, 3);
}
crates/clawhdf5-agent/tests/search_options.rs
+344
View File
@@ -0,0 +1,344 @@
//! `HDF5Memory::search` with `SearchOptions`: source filtering, re-ranking and
//! confidence rejection in the store's own search path.
use std::collections::HashSet;
use clawhdf5_agent::confidence::ConfidenceConfig;
use clawhdf5_agent::reranker::ReRankConfig;
use clawhdf5_agent::{AgentMemory, HDF5Memory, MemoryConfig, MemoryEntry, SearchOptions, hybrid};
use tempfile::TempDir;
const DIM: usize = 32;
const N: usize = 3000;
const CLUSTERS: usize = 20;
struct Rng(u64);
impl Rng {
fn next(&mut self) -> u64 {
self.0 = self.0.wrapping_add(0x9E37_79B9_7F4A_7C15);
let mut z = self.0;
z = (z ^ (z >> 30)).wrapping_mul(0xBF58_476D_1CE4_E5B9);
z = (z ^ (z >> 27)).wrapping_mul(0x94D0_49BB_1331_11EB);
z ^ (z >> 31)
}
fn unit(&mut self) -> f32 {
(self.next() >> 40) as f32 / (1u64 << 24) as f32 - 0.5
}
}
fn normalize(v: &mut [f32]) {
let n = v.iter().map(|x| x * x).sum::<f32>().sqrt();
v.iter_mut().for_each(|x| *x /= n);
}
struct Data {
vectors: Vec<Vec<f32>>,
cluster: Vec<usize>,
centres: Vec<Vec<f32>>,
}
fn data() -> Data {
let mut rng = Rng(42);
let centres: Vec<Vec<f32>> = (0..CLUSTERS)
.map(|_| {
let mut c: Vec<f32> = (0..DIM).map(|_| rng.unit()).collect();
normalize(&mut c);
c
})
.collect();
let mut vectors = Vec::new();
let mut cluster = Vec::new();
for i in 0..N {
let c = i % CLUSTERS;
let mut v: Vec<f32> = centres[c].iter().map(|x| x + rng.unit() * 0.3).collect();
normalize(&mut v);
vectors.push(v);
cluster.push(c);
}
Data {
vectors,
cluster,
centres,
}
}
/// Channel of record `i` for a filter keeping `percent`% of the store at
/// random (independent of the vectors).
fn random_channel(i: usize, rng_seed: u64, percent: u64) -> String {
let mut r = Rng(rng_seed ^ (i as u64 * 7919));
if r.next() % 100 < percent {
"keep".into()
} else {
"other".into()
}
}
fn build(data: &Data, channel: impl Fn(usize) -> String) -> (TempDir, HDF5Memory) {
let dir = TempDir::new().unwrap();
let mut cfg = MemoryConfig::new(dir.path().join("s.h5"), "agent", DIM);
cfg.hebbian_boost = 0.0; // every query sees the same store
let mut m = HDF5Memory::create(cfg).unwrap();
let entries = data
.vectors
.iter()
.enumerate()
.map(|(i, v)| MemoryEntry {
chunk: format!("record {i} cluster {}", data.cluster[i]),
embedding: v.clone(),
source_channel: channel(i),
timestamp: i as f64,
session_id: "s".into(),
tags: format!("t{i}"),
})
.collect();
m.save_batch(entries).unwrap();
(dir, m)
}
/// Exact top-k by cosine among the records `allowed` keeps.
fn exact_top(data: &Data, q: &[f32], k: usize, allowed: impl Fn(usize) -> bool) -> Vec<usize> {
let mut s: Vec<(usize, f32)> = (0..N)
.filter(|&i| allowed(i))
.map(|i| (i, data.vectors[i].iter().zip(q).map(|(a, b)| a * b).sum()))
.collect();
s.sort_by(|a, b| b.1.total_cmp(&a.1).then(a.0.cmp(&b.0)));
s.into_iter().take(k).map(|(i, _)| i).collect()
}
fn query(data: &Data, i: usize) -> Vec<f32> {
let mut rng = Rng(1000 + i as u64);
let mut q: Vec<f32> = data.centres[i % CLUSTERS]
.iter()
.map(|x| x + rng.unit() * 0.3)
.collect();
normalize(&mut q);
q
}
fn vector_only(k: usize) -> SearchOptions {
SearchOptions::new(k).with_fusion(hybrid::Fusion::Weighted {
vector: 1.0,
keyword: 0.0,
})
}
#[test]
fn source_filter_returns_only_allowed_records_and_a_full_page() {
let d = data();
// At N = 3000 and k = 10 the index serves a filter only when that is
// cheaper than scanning the allowed records: pool = 80 * N / allowed
// candidates at ~M = 16 distances each, against `allowed` distances. So
// 90% goes through the index, 50% and 1% to the exact scan.
for percent in [90, 50, 1] {
let (_dir, mut m) = build(&d, |i| random_channel(i, 5, percent));
let allowed = |i: usize| random_channel(i, 5, percent) == "keep";
let mut hits = 0;
for qi in 0..40 {
let q = query(&d, qi);
let got = m.search(&q, "", &vector_only(10).with_sources(["keep"]));
assert_eq!(got.len(), 10, "{percent}%: short page");
assert!(got.iter().all(|r| r.source_channel == "keep"));
let want: HashSet<usize> = exact_top(&d, &q, 10, allowed).into_iter().collect();
hits += got.iter().filter(|r| want.contains(&r.index)).count();
}
let recall = hits as f64 / 400.0;
let floor = if percent == 90 { 0.95 } else { 1.0 };
assert!(recall >= floor, "{percent}%: recall@10 {recall}");
}
}
#[test]
fn filter_away_from_the_query_falls_back_to_an_exact_scan() {
// Channel = cluster, and the filter keeps two clusters (10% of the
// store) that are not the query's: the index's neighbourhood of the
// query holds none of them. The search must still return the exact
// top 10 among the allowed records, not a short or empty page.
let d = data();
let (_dir, mut m) = build(&d, |i| format!("c{}", d.cluster[i]));
for qi in 0..20 {
let q = query(&d, qi);
let a = format!("c{}", (qi + 7) % CLUSTERS);
let b = format!("c{}", (qi + 13) % CLUSTERS);
let got: Vec<usize> = m
.search(
&q,
"",
&vector_only(10).with_sources([a.clone(), b.clone()]),
)
.iter()
.map(|r| r.index)
.collect();
let want = exact_top(&d, &q, 10, |i| {
let c = format!("c{}", d.cluster[i]);
c == a || c == b
});
assert_eq!(got, want, "query {qi}");
}
}
#[test]
fn filter_edge_cases() {
let d = data();
let (_dir, mut m) = build(&d, |i| random_channel(i, 9, 50));
let q = query(&d, 0);
assert!(
m.search(
&q,
"cluster",
&SearchOptions::new(10).with_sources(Vec::<String>::new())
)
.is_empty()
);
assert!(
m.search(
&q,
"cluster",
&SearchOptions::new(10).with_sources(["nope"])
)
.is_empty()
);
// Keyword matches from other channels are filtered too.
let got = m.search(
&q,
"record cluster",
&SearchOptions::new(50).with_sources(["keep"]),
);
assert_eq!(got.len(), 50);
assert!(got.iter().all(|r| r.source_channel == "keep"));
// Deleted records never come back, filtered or not.
let first = got[0].index;
m.delete(first).unwrap();
let again = m.search(
&q,
"record cluster",
&SearchOptions::new(50).with_sources(["keep"]),
);
assert!(again.iter().all(|r| r.index != first));
}
#[test]
fn plain_options_equal_hybrid_search_with() {
// Two identical stores, so neither query sees the other's boosts.
let d = data();
let (_a, mut a) = build(&d, |i| random_channel(i, 3, 50));
let (_b, mut b) = build(&d, |i| random_channel(i, 3, 50));
for qi in 0..10 {
let q = query(&d, qi);
let x: Vec<(usize, u32)> = a
.search(&q, "record cluster 3", &SearchOptions::new(10))
.iter()
.map(|r| (r.index, r.score.to_bits()))
.collect();
let y: Vec<(usize, u32)> = b
.hybrid_search_with(&q, "record cluster 3", hybrid::DEFAULT_FUSION, 10)
.iter()
.map(|r| (r.index, r.score.to_bits()))
.collect();
assert_eq!(x, y);
}
}
fn small_store(entries: &[(&str, &str, f64)]) -> (TempDir, HDF5Memory) {
let dir = TempDir::new().unwrap();
let mut m = HDF5Memory::create(MemoryConfig::new(dir.path().join("r.h5"), "a", 4)).unwrap();
m.save_batch(
entries
.iter()
.map(|(chunk, channel, ts)| MemoryEntry {
chunk: chunk.to_string(),
embedding: vec![1.0, 0.0, 0.0, 0.0],
source_channel: channel.to_string(),
timestamp: *ts,
session_id: "s".into(),
tags: String::new(),
})
.collect(),
)
.unwrap();
(dir, m)
}
#[test]
fn rerank_breaks_relevance_ties_by_recency() {
// Identical text and vectors, so retrieval ties; re-ranking must put the
// newer record first and report the combined score.
let now = 1_000_000.0;
let (_d, mut m) = small_store(&[
("user prefers dark mode", "chat", now - 30.0 * 86_400.0),
("user prefers dark mode", "chat", now - 60.0),
]);
let q = [1.0, 0.0, 0.0, 0.0];
let plain = m.search(&q, "dark mode", &SearchOptions::new(2));
assert_eq!(plain[0].index, 0, "ties break by index without re-ranking");
let reranked = m.search(
&q,
"dark mode",
&SearchOptions::new(2)
.with_rerank(ReRankConfig::default())
.at_time(now),
);
assert_eq!(reranked[0].index, 1);
assert!(reranked[0].score > reranked[1].score);
assert_ne!(reranked[0].score.to_bits(), plain[0].score.to_bits());
}
#[test]
fn confidence_rejects_when_nothing_is_good_enough() {
let (_d, mut m) = small_store(&[("alpha", "chat", 0.0), ("beta", "chat", 0.0)]);
let q = [1.0, 0.0, 0.0, 0.0];
let strict = ConfidenceConfig {
min_score: 10.0,
..ConfidenceConfig::default()
};
assert!(
m.search(&q, "alpha", &SearchOptions::new(2).with_confidence(strict))
.is_empty()
);
let lenient = ConfidenceConfig {
min_score: 0.0,
min_gap: f32::INFINITY,
max_results: 1,
};
assert_eq!(
m.search(&q, "alpha", &SearchOptions::new(2).with_confidence(lenient))
.len(),
1
);
}
#[test]
fn only_returned_results_are_reinforced() {
// With re-ranking, a pool of max(3k, 10) candidates is retrieved; only
// the k returned should gain activation.
let d = data();
let dir = TempDir::new().unwrap();
let path = dir.path().join("h.h5");
let mut m = HDF5Memory::create(MemoryConfig::new(path, "a", DIM)).unwrap();
m.save_batch(
(0..200)
.map(|i| MemoryEntry {
chunk: format!("record {i}"),
embedding: d.vectors[i].clone(),
source_channel: "chat".into(),
timestamp: i as f64,
session_id: "s".into(),
tags: String::new(),
})
.collect(),
)
.unwrap();
let q = query(&d, 0);
let got = m.search(
&q,
"record",
&SearchOptions::new(3).with_rerank(ReRankConfig::default()),
);
assert_eq!(got.len(), 3);
let returned: HashSet<usize> = got.iter().map(|r| r.index).collect();
// A second plain search reports each record's current activation.
let all = m.search(&q, "record", &SearchOptions::new(200));
for r in &all {
let boosted = r.activation > 1.0;
assert_eq!(boosted, returned.contains(&r.index), "record {}", r.index);
}
}
crates/clawhdf5-agent/tests/signed_store.rs
+330
View File
@@ -0,0 +1,330 @@
//! Ed25519-signed checkpoints: `HDF5Memory::set_signing_key` and
//! `HDF5Memory::verify`.
use std::path::Path;
use clawhdf5_agent::signing::{SigningKey, VerifyReport, VerifyingKey};
use clawhdf5_agent::storage;
use clawhdf5_agent::{AgentMemory, HDF5Memory, MemoryConfig, MemoryEntry, MemoryError, schema};
use tempfile::TempDir;
const DIM: usize = 16;
fn key(seed: u8) -> SigningKey {
SigningKey::from_bytes(&[seed; 32])
}
fn entry(i: usize, chunk: &str) -> MemoryEntry {
MemoryEntry {
chunk: chunk.to_string(),
embedding: (0..DIM)
.map(|j| ((i * DIM + j) as f32 * 0.37).sin())
.collect(),
source_channel: "chat".into(),
timestamp: 1_700_000_000.0 + i as f64,
session_id: format!("s{}", i % 3),
tags: format!("t{i}"),
}
}
/// Awkward strings on purpose: they must hash the same after a round trip.
const TEXTS: [&str; 6] = [
"plain text",
"ünïcödé — 日本語 🙂",
"",
"trailing spaces ",
"tab\tand\nnewline",
"x",
];
fn signed_store(dir: &TempDir, float16: bool, k: &SigningKey) -> std::path::PathBuf {
let mut cfg = MemoryConfig::new(dir.path().join("s.h5"), "agent", DIM);
cfg.float16 = float16;
let path = cfg.path.clone();
let mut m = HDF5Memory::create(cfg).unwrap();
m.set_signing_key(k.clone());
let entries = (0..30).map(|i| entry(i, TEXTS[i % TEXTS.len()])).collect();
m.save_batch(entries).unwrap();
// Some graph and a deleted record, so every part of the manifest is used.
let a = m.knowledge_mut().add_entity("Alice", "person", 0);
let b = m.knowledge_mut().add_entity("Acme", "org", -1);
m.knowledge_mut().add_relation(a, b, "works_at", 0.75);
m.sessions_mut()
.add_at("s0", 0, 9, "chat", "first session", 1_700_000_000.0);
m.delete(4).unwrap();
m.flush_wal().unwrap();
path
}
fn verify(path: &Path, k: &SigningKey) -> VerifyReport {
HDF5Memory::verify(path, &k.verifying_key()).unwrap()
}
#[test]
fn a_signed_store_verifies_through_reopen_and_checkpoint_cycles() {
for float16 in [true, false] {
let dir = TempDir::new().unwrap();
let k = key(7);
let path = signed_store(&dir, float16, &k);
let r = verify(&path, &k);
assert!(r.is_valid(), "float16={float16}: {r:?}");
assert_eq!(r.public_key, Some(k.verifying_key().to_bytes()));
assert_eq!(r.record_count, 30);
assert!(r.changed_records.is_empty());
// Reopen, change nothing, checkpoint again (with the key): still valid.
for _ in 0..3 {
let mut m = HDF5Memory::open(&path).unwrap();
assert!(m.is_signed());
m.set_signing_key(k.clone());
m.flush_wal().unwrap();
drop(m);
assert!(verify(&path, &k).is_valid());
}
// And after real changes, re-signed.
let mut m = HDF5Memory::open(&path).unwrap();
m.set_signing_key(k.clone());
m.save(entry(99, "added later")).unwrap();
m.hybrid_search(&entry(1, "").embedding, "text", 0.4, 0.6, 5);
m.flush_wal().unwrap();
drop(m);
let r = verify(&path, &k);
assert!(r.is_valid(), "{r:?}");
assert_eq!(r.record_count, 31);
}
}
#[test]
fn a_signed_store_refuses_to_checkpoint_without_its_key() {
let dir = TempDir::new().unwrap();
let k = key(1);
let path = signed_store(&dir, true, &k);
let mut m = HDF5Memory::open(&path).unwrap();
m.save(entry(50, "pending")).unwrap();
match m.flush_wal() {
Err(MemoryError::SigningKeyRequired(msg)) => assert!(msg.contains("signed"), "{msg}"),
other => panic!("expected SigningKeyRequired, got {other:?}"),
}
// The file is untouched and still valid; the save is still in the WAL.
let r = verify(&path, &k);
assert!(r.is_valid());
assert_eq!(r.wal_entries_unsigned, 1);
// Supplying the key lets the checkpoint through, signed.
m.set_signing_key(k.clone());
m.flush_wal().unwrap();
drop(m);
let r = verify(&path, &k);
assert!(r.is_valid());
assert_eq!((r.record_count, r.wal_entries_unsigned), (31, 0));
// Removing the signature on purpose writes it unsigned.
let mut m = HDF5Memory::open(&path).unwrap();
m.remove_signature();
m.flush_wal().unwrap();
drop(m);
let r = verify(&path, &k);
assert!(!r.signed && !r.is_valid());
assert!(!HDF5Memory::open(&path).unwrap().is_signed());
}
#[test]
fn the_wrong_key_does_not_verify_and_a_new_key_re_signs() {
let dir = TempDir::new().unwrap();
let (a, b) = (key(1), key(2));
let path = signed_store(&dir, true, &a);
let r = verify(&path, &b);
assert!(r.signed && !r.key_matches && !r.signature_valid && !r.is_valid());
let mut m = HDF5Memory::open(&path).unwrap();
m.set_signing_key(b.clone());
m.flush_wal().unwrap();
drop(m);
assert!(verify(&path, &b).is_valid());
assert!(!verify(&path, &a).is_valid());
}
/// Rewrite the store with changed contents but the *old* signature — what
/// someone with write access to the file, but not the key, can do.
fn tamper(path: &Path, change: impl FnOnce(&mut Tampered)) {
let file = clawhdf5::File::open(path).unwrap();
let (config, cache, sessions, knowledge) = schema::validate_and_load(&file).unwrap();
let checkpoint = schema::read_checkpoint_meta(&file);
let signature = schema::read_signature(&file).unwrap().unwrap();
drop(file);
let mut t = Tampered {
config,
cache,
sessions,
knowledge,
};
change(&mut t);
storage::write_to_disk_signed(
path,
&t.config,
&t.cache,
&t.sessions,
&t.knowledge,
&checkpoint,
Some(&signature),
)
.unwrap();
}
struct Tampered {
config: MemoryConfig,
cache: clawhdf5_agent::cache::MemoryCache,
sessions: clawhdf5_agent::SessionCache,
knowledge: clawhdf5_agent::knowledge::KnowledgeCache,
}
#[test]
fn every_kind_of_edit_is_detected_and_located() {
let k = key(3);
type Edit = Box<dyn FnOnce(&mut Tampered)>;
type Case = (&'static str, Edit, fn(&VerifyReport) -> bool);
let cases: Vec<Case> = vec![
(
"record text",
Box::new(|t: &mut Tampered| t.cache.chunks[7] = "rewritten".into()),
|r| !r.records_match && r.changed_records == vec![7],
),
(
"one embedding value",
Box::new(|t: &mut Tampered| {
let mut e = t.cache.embeddings[12].to_vec();
e[3] = 0.5;
t.cache.embeddings.set(12, &e);
}),
|r| r.changed_records == vec![12],
),
(
"undelete",
Box::new(|t: &mut Tampered| t.cache.tombstones[4] = 0),
|r| r.changed_records == vec![4],
),
(
"timestamp",
Box::new(|t: &mut Tampered| t.cache.timestamps[20] += 1.0),
|r| r.changed_records == vec![20],
),
(
"record appended",
Box::new(|t: &mut Tampered| {
t.cache.push(
"new".into(),
vec![0.1; DIM],
"x".into(),
1.0,
"s".into(),
"".into(),
);
}),
|r| !r.records_match && r.changed_records == vec![30] && r.record_count == 31,
),
(
"setting",
Box::new(|t: &mut Tampered| t.config.agent_id = "someone-else".into()),
|r| !r.settings_match && r.records_match,
),
(
"session summary",
Box::new(|t: &mut Tampered| t.sessions.summaries[0] = "edited".into()),
|r| !r.sessions_match && r.records_match,
),
(
"graph edge",
Box::new(|t: &mut Tampered| t.knowledge.relations[0].weight = 1.0),
|r| !r.graph_match && r.records_match,
),
];
for (name, edit, check) in cases {
let dir = TempDir::new().unwrap();
let path = signed_store(&dir, true, &k);
tamper(&path, edit);
let r = verify(&path, &k);
assert!(
r.signed && r.key_matches && r.signature_valid,
"{name}: {r:?}"
);
assert!(!r.is_valid(), "{name}: edit not detected: {r:?}");
assert!(check(&r), "{name}: {r:?}");
}
}
#[test]
fn a_forged_manifest_fails_the_signature() {
// Recomputing the hashes for tampered contents does not help without the
// key: the signature no longer matches the manifest.
let dir = TempDir::new().unwrap();
let k = key(5);
let path = signed_store(&dir, true, &k);
let file = clawhdf5::File::open(&path).unwrap();
let (config, mut cache, sessions, knowledge) = schema::validate_and_load(&file).unwrap();
let checkpoint = schema::read_checkpoint_meta(&file);
let mut sig = schema::read_signature(&file).unwrap().unwrap();
drop(file);
cache.chunks[0] = "forged".into();
// Re-sign with an attacker key, then splice the victim's public key back.
let forged = clawhdf5_agent::signing::sign(
&key(66),
&config,
&cache,
&sessions,
&knowledge,
checkpoint.wal_applied,
);
sig.manifest = forged.manifest;
sig.record_hashes = forged.record_hashes;
storage::write_to_disk_signed(
&path,
&config,
&cache,
&sessions,
&knowledge,
&checkpoint,
Some(&sig),
)
.unwrap();
let r = verify(&path, &k);
assert!(
r.key_matches && !r.signature_valid && !r.is_valid(),
"{r:?}"
);
}
#[test]
fn an_unsigned_store_reports_unsigned() {
let dir = TempDir::new().unwrap();
let mut m = HDF5Memory::create(MemoryConfig::new(dir.path().join("u.h5"), "a", DIM)).unwrap();
m.save_batch(vec![entry(0, "hello")]).unwrap();
drop(m);
let r = HDF5Memory::verify(&dir.path().join("u.h5"), &VerifyingKey::from(&key(1))).unwrap();
assert!(!r.signed && !r.is_valid());
assert_eq!(r.record_count, 1);
}
#[test]
fn nul_bytes_in_text_still_verify() {
// Strings are stored null-padded; the hash must follow what a reopened
// store actually holds, or an untouched store would fail to verify.
let dir = TempDir::new().unwrap();
let k = key(9);
let mut m = HDF5Memory::create(MemoryConfig::new(dir.path().join("n.h5"), "a", DIM)).unwrap();
m.set_signing_key(k.clone());
m.save_batch(vec![
entry(0, "inner\0nul"),
entry(1, "trailing nul\0"),
entry(2, "\0leading"),
])
.unwrap();
drop(m);
let r = verify(&dir.path().join("n.h5"), &k);
assert!(r.is_valid(), "{r:?}");
let m = HDF5Memory::open(&dir.path().join("n.h5")).unwrap();
eprintln!(
"reloaded: {:?}",
(0..3).map(|i| m.get_chunk(i)).collect::<Vec<_>>()
);
}
crates/clawhdf5-android/Cargo.toml
+2 -1
View File
@@ -1,7 +1,8 @@
[package]
name = "clawhdf5-android"
version = "2.4.0"
version = "2.7.0"
edition = "2024"
rust-version.workspace = true
description = "Android JNI bridge for edgehdf5-memory HDF5 backend"
license = "MIT"
crates/clawhdf5-ann/Cargo.toml
+5 -4
View File
@@ -1,7 +1,8 @@
[package]
name = "clawhdf5-ann"
version = "2.4.0"
version = "2.7.0"
edition = "2024"
rust-version.workspace = true
description = "HNSW approximate nearest neighbor index stored as HDF5"
license = "MIT"
repository = "https://git.redclaw.dev/quantumclaw/clawhdf5"
@@ -10,9 +11,9 @@ keywords = ["hdf5", "ann", "hnsw", "nearest-neighbor"]
categories = ["algorithms", "science"]
[dependencies]
clawhdf5-format = { path = "../clawhdf5-format", version = "2.4.0" }
clawhdf5-io = { path = "../clawhdf5-io", version = "2.4.0" }
clawhdf5-accel = { path = "../clawhdf5-accel", version = "2.4.0" }
clawhdf5-format = { path = "../clawhdf5-format", version = "2.7.0" }
clawhdf5-io = { path = "../clawhdf5-io", version = "2.7.0" }
clawhdf5-accel = { path = "../clawhdf5-accel", version = "2.7.0" }
rayon = { version = "1", optional = true }
[features]
crates/clawhdf5-ann/src/hnsw.rs
+659 -121
View File
File diff suppressed because it is too large Load Diff
crates/clawhdf5-ann/src/lib.rs
+1 -1
View File
@@ -5,4 +5,4 @@
mod hnsw;
pub use hnsw::{DistanceMetric, HnswIndex};
pub use hnsw::{DistanceMetric, HnswIndex, Storage};
crates/clawhdf5-bench/Cargo.toml
+16 -1
View File
@@ -1,7 +1,8 @@
[package]
name = "clawhdf5-bench"
version = "2.4.0"
version = "2.7.0"
edition = "2024"
rust-version.workspace = true
description = "Benchmark harnesses for clawhdf5-agent (Track 8)"
license = "MIT"
@@ -13,6 +14,10 @@ path = "src/bin/longmemeval_bench.rs"
name = "memory_arena"
path = "src/bin/memory_arena.rs"
[[bin]]
name = "read_harness"
path = "src/bin/read_harness.rs"
[[bin]]
name = "search_harness"
path = "src/bin/search_harness.rs"
@@ -29,6 +34,10 @@ path = "src/bin/consolidation_efficiency.rs"
name = "ephemeral_perf"
path = "src/bin/ephemeral_perf.rs"
[[bin]]
name = "concurrent_read"
path = "src/bin/concurrent_read.rs"
[[bin]]
name = "mpi_io_bench"
path = "src/bin/mpi_io_bench.rs"
@@ -53,10 +62,16 @@ harness = false
[dependencies]
clawhdf5-agent = { path = "../clawhdf5-agent" }
clawhdf5-ann = { path = "../clawhdf5-ann" }
clawhdf5 = { path = "../clawhdf5" }
clawhdf5-format = { path = "../clawhdf5-format" }
clawhdf5-io = { path = "../clawhdf5-io" }
mpi = { version = "0.8", optional = true }
serde = { workspace = true }
serde_json = "1"
# concurrent_read: size the decode pool (--decode-threads) and evict files
# from the page cache (--cold, posix_fadvise). Both pure Rust / bindings only.
rayon = "1"
libc = "0.2"
tempfile = { workspace = true }
# Optional: libhdf5 C wrapper for side-by-side comparison (requires system libhdf5).
# Enable with: cargo bench -p clawhdf5-bench --features libhdf5-compare
crates/clawhdf5-bench/scripts/__pycache__/compare_concurrent_read.cpython-314.pyc
BIN
View File
Binary file not shown.
crates/clawhdf5-bench/scripts/__pycache__/concurrent_read_h5py.cpython-314.pyc
BIN
View File
Binary file not shown.
crates/clawhdf5-bench/scripts/compare_concurrent_read.py
+70
View File
@@ -0,0 +1,70 @@
#!/usr/bin/env python3
"""Tabulate concurrent_read JSON results (clawhdf5, h5py threads/processes).
python compare_concurrent_read.py clawhdf5.json h5py-threads.json h5py-procs.json
Prints one Markdown table: for each layout, mode and thread count, every
tool's MB/s and scaling efficiency, and the first file's MB/s relative to each
of the others. Refuses to compare runs whose workload parameters differ.
"""
import json
import sys
COMPARED = ("datasets", "rows", "cols", "chunk", "deflate_level", "slab", "slabs", "seed")
def main(paths):
if len(paths) < 2:
sys.exit(__doc__)
docs = []
for p in paths:
with open(p) as fh:
docs.append(json.load(fh))
ref = docs[0]
for d, p in zip(docs[1:], paths[1:]):
diff = [k for k in COMPARED if d["params"].get(k) != ref["params"].get(k)]
if diff:
sys.exit(f"{p}: workload differs from {paths[0]} in {', '.join(diff)}")
if d["cache"] != ref["cache"]:
print(f"warning: {p} ran {d['cache']!r}, {paths[0]} ran {ref['cache']!r}",
file=sys.stderr)
if d.get("host") != ref.get("host"):
print(f"warning: {p} ran on {d.get('host')}, {paths[0]} on {ref.get('host')}",
file=sys.stderr)
names = [d["tool"] for d in docs]
for d in docs:
extra = f", HDF5 {d['hdf5_version']}" if "hdf5_version" in d else ""
print(f"- {d['tool']} {d['version']}{extra}: host {d.get('host')}, "
f"{d.get('cpus')} CPUs, cache {d['cache']}, decode threads per read "
f"{d.get('decode_threads')}")
p = ref["params"]
print(f"\n{p['datasets']} datasets of {p['rows']} x {p['cols']} f32, chunks "
f"{p['chunk'][0]} x {p['chunk'][1]} (deflate {p['deflate_level']}); "
f"`same`: {p['slabs']} slabs of {p['slab']} x {p['slab']}\n")
index = [{(r["layout"], r["mode"], r["threads"]): r for r in d["results"]} for d in docs]
keys = [(r["layout"], r["mode"], r["threads"]) for r in ref["results"]]
head = ["layout", "mode", "threads"]
head += [f"{n} MB/s (eff)" for n in names]
head += [f"{names[0]} / {n}" for n in names[1:]]
print("| " + " | ".join(head) + " |")
print("|---|---|" + "---:|" * (len(head) - 2))
for key in keys:
cells = [key[0], key[1], str(key[2])]
rs = [ix.get(key) for ix in index]
for r in rs:
if r is None:
cells.append("-")
else:
eff = "-" if r["efficiency"] is None else f"{r['efficiency']:.2f}"
cells.append(f"{r['mb_s']:.0f} ({eff})")
for r in rs[1:]:
cells.append("-" if r is None else f"{rs[0]['mb_s'] / r['mb_s']:.2f}x")
print("| " + " | ".join(cells) + " |")
if __name__ == "__main__":
main(sys.argv[1:])
crates/clawhdf5-bench/scripts/concurrent_read_h5py.py
+265
View File
@@ -0,0 +1,265 @@
#!/usr/bin/env python3
"""The concurrent_read workload with h5py, on the files concurrent_read wrote.
libhdf5 serialises every API call under one global lock, and h5py holds its
own global lock around every call as well, so h5py *threads* cannot decode in
parallel. h5py users scale with *processes* instead; ``--executor processes``
measures that (each worker opens the file itself).
The workload mirrors ``crates/clawhdf5-bench/src/bin/concurrent_read.rs``:
* ``distinct``: every dataset read in full once per repetition; worker ``t``
of ``T`` reads datasets ``t, t + T, ...``.
* ``same``: ``--slabs`` random ``--slab`` x ``--slab`` hyperslabs of ``d00``
(slab ``j`` to worker ``j % T``), offsets from the same splitmix64 stream.
Each worker times itself from a start barrier; a repetition spans the earliest
start to the latest finish (CLOCK_MONOTONIC, comparable across processes).
Threads share one ``h5py.File`` per repetition; process workers open the file
inside the timed region (a few ms against reads of many MiB).
Generate the files first with the Rust harness (it writes ``manifest.json``),
then, for example::
python concurrent_read_h5py.py --dir DIR --executor threads --json h5py-threads.json
python concurrent_read_h5py.py --dir DIR --executor processes --json h5py-procs.json
"""
import argparse
import json
import multiprocessing as mp
import os
import platform
import socket
import sys
import threading
import time
import h5py
import numpy as np
M64 = (1 << 64) - 1
def splitmix64(state):
"""Return (new_state, value); the same stream as the Rust harness."""
state = (state + 0x9E3779B97F4A7C15) & M64
z = state
z = ((z ^ (z >> 30)) * 0xBF58476D1CE4E5B9) & M64
z = ((z ^ (z >> 27)) * 0x94D049BB133111EB) & M64
return state, z ^ (z >> 31)
def value(k, i):
"""Element i (row-major) of dataset k, exactly as concurrent_read writes it."""
_, noise = splitmix64(i ^ (k << 40))
return np.float32((((i >> 6) % 16384) + k) + (noise & 0xFF) / 256.0)
def slab_offsets(seed, count, rows, cols, slab):
s = seed
out = []
for _ in range(count):
s, r = splitmix64(s)
s, c = splitmix64(s)
out.append((r % (rows - slab + 1), c % (cols - slab + 1)))
return out
def now():
return time.clock_gettime(time.CLOCK_MONOTONIC)
def work(f, mode, t, threads, m, slabs, slab, verify):
"""Worker t's share of one repetition on an open h5py.File."""
n = m["rows"] * m["cols"]
if mode == "distinct":
for k in range(t, m["datasets"], threads):
got = f[f"d{k:02d}"][...]
assert got.size == n
if verify:
flat = got.reshape(-1)
for i in (0, n // 3, n - 1):
assert flat[i] == value(k, i), f"d{k:02d}[{i}]"
else:
ds = f["d00"]
cols = m["cols"]
for r, c in slabs[t::threads]:
got = ds[r : r + slab, c : c + slab]
assert got.shape == (slab, slab)
if verify:
assert got[0, 0] == value(0, r * cols + c)
last = (r + slab - 1) * cols + c + slab - 1
assert got[-1, -1] == value(0, last)
# ----- process workers ------------------------------------------------------
_barrier = None
def _init(barrier):
global _barrier
_barrier = barrier
def _proc_task(task):
path, mode, t, threads, m, slabs, slab = task
_barrier.wait()
start = now()
with h5py.File(path, "r") as f:
work(f, mode, t, threads, m, slabs, slab, False)
return start, now()
def _noop(_):
return os.getpid()
def run_threads(path, mode, threads, m, slabs, slab):
spans = [None] * threads
barrier = threading.Barrier(threads)
with h5py.File(path, "r") as f:
def body(t):
barrier.wait()
start = now()
work(f, mode, t, threads, m, slabs, slab, False)
spans[t] = (start, now())
ts = [threading.Thread(target=body, args=(t,)) for t in range(threads)]
for th in ts:
th.start()
for th in ts:
th.join()
return max(e for _, e in spans) - min(s for s, _ in spans)
def run_processes(pool, path, mode, threads, m, slabs, slab):
tasks = [(path, mode, t, threads, m, slabs, slab) for t in range(threads)]
# One task per worker: each blocks in the barrier until all T have
# started, so no worker can take a second task.
spans = pool.map(_proc_task, tasks, chunksize=1)
return max(e for _, e in spans) - min(s for s, _ in spans)
def warm(path):
with open(path, "rb") as fh:
while fh.read(1 << 24):
pass
def evict(path):
fd = os.open(path, os.O_RDONLY)
try:
os.posix_fadvise(fd, 0, 0, os.POSIX_FADV_DONTNEED)
finally:
os.close(fd)
def main():
ap = argparse.ArgumentParser(description=__doc__.split("\n\n")[0])
ap.add_argument("--dir", default="concurrent-read-data")
ap.add_argument("--executor", choices=["threads", "processes"], default="threads")
ap.add_argument("--threads", default="1,2,4,8,16")
ap.add_argument("--reps", type=int, default=3)
ap.add_argument("--slab", type=int, default=256)
ap.add_argument("--slabs", type=int, default=1024)
ap.add_argument("--seed", type=int, default=42)
ap.add_argument("--cold", action="store_true")
ap.add_argument("--modes", default="distinct,same")
ap.add_argument("--layouts", default="deflate,contiguous")
ap.add_argument("--json")
a = ap.parse_args()
# The Rust harness pins this value (splitmix64_reference).
assert splitmix64(42)[1] == 0xBDD732262FEB6E95, "splitmix64 port is wrong"
try:
with open(os.path.join(a.dir, "manifest.json")) as fh:
m = json.load(fh)
except FileNotFoundError:
sys.exit(f"{a.dir}/manifest.json not found: generate the files with "
"`cargo run --release -p clawhdf5-bench --bin concurrent_read -- --dir ...` first")
threads_list = [int(x) for x in a.threads.split(",")]
modes = a.modes.split(",")
layouts = a.layouts.split(",")
if a.slab < 1 or a.slab > min(m["rows"], m["cols"]):
sys.exit(f"--slab must be 1..={min(m['rows'], m['cols'])}")
files = dict(m["files"])
slabs = slab_offsets(a.seed, a.slabs, m["rows"], m["cols"], a.slab)
dataset_bytes = m["rows"] * m["cols"] * 4
tool = f"h5py-{a.executor}"
ctx = mp.get_context("spawn") # never fork a process holding HDF5 state
pools = {}
if a.executor == "processes":
for t in threads_list:
pool = ctx.Pool(t, initializer=_init, initargs=(ctx.Barrier(t),))
pool.map(_noop, range(t)) # start the workers outside the timing
pools[t] = pool
rows = []
print("| layout | mode | threads | MB/s | efficiency | median s |")
print("|---|---|---:|---:|---:|---:|")
try:
for layout in layouts:
path = os.path.join(a.dir, files[layout])
if not a.cold:
warm(path)
for mode in modes:
with h5py.File(path, "r") as f: # untimed, checked pass
work(f, mode, 0, 1, m, slabs, a.slab, True)
nbytes = (dataset_bytes * m["datasets"] if mode == "distinct"
else a.slab * a.slab * 4 * a.slabs)
base = None
for t in threads_list:
times = []
for _ in range(a.reps):
if a.cold:
evict(path)
if a.executor == "threads":
times.append(run_threads(path, mode, t, m, slabs, a.slab))
else:
times.append(run_processes(pools[t], path, mode, t, m, slabs, a.slab))
med = sorted(times)[len(times) // 2]
mb_s = nbytes / (1 << 20) / med
if t == 1:
base = mb_s
eff = mb_s / (t * base) if base else None
print(f"| {layout} | {mode} | {t} | {mb_s:.0f} | "
f"{'-' if eff is None else f'{eff:.2f}'} | {med:.4f} |")
rows.append({
"layout": layout, "mode": mode, "threads": t, "bytes": nbytes,
"times_s": times, "median_s": med, "mb_s": mb_s, "efficiency": eff,
})
finally:
for pool in pools.values():
pool.terminate()
if a.json:
doc = {
"tool": tool,
"version": h5py.__version__,
"hdf5_version": h5py.version.hdf5_version,
"python": platform.python_version(),
"host": socket.gethostname(),
"cpus": os.cpu_count(),
"unix_time": int(time.time()),
"cache": ("cold (posix_fadvise DONTNEED before each repetition)"
if a.cold else "warm"),
"decode_threads": 1,
"params": {
"datasets": m["datasets"], "rows": m["rows"], "cols": m["cols"],
"chunk": m["chunk"], "deflate_level": m["deflate_level"],
"mib": dataset_bytes // (1 << 20), "slab": a.slab, "slabs": a.slabs,
"seed": a.seed, "reps": a.reps, "dir": a.dir,
},
"results": rows,
}
with open(a.json, "w") as fh:
json.dump(doc, fh, indent=2)
if __name__ == "__main__":
main()
crates/clawhdf5-bench/src/bin/concurrent_read.rs
+523
View File
@@ -0,0 +1,523 @@
//! Concurrent-read harness: how does decoded read throughput scale with the
//! number of threads reading one open file?
//!
//! libhdf5 (threadsafe build) serialises every API call under one global
//! mutex, and h5py holds it too, so threads cannot decode in parallel there.
//! A clawhdf5 [`File`] is `Send + Sync`; this harness measures what that buys.
//! `crates/clawhdf5-bench/scripts/concurrent_read_h5py.py` runs the same
//! workload on the same files with h5py (threads, and processes), and
//! `compare_concurrent_read.py` tabulates the JSON both write.
//!
//! Files (generated on first use, reused while `manifest.json` matches):
//!
//! * `<dir>/deflate.h5`: `--datasets` datasets `d00`, `d01`, ... of `f32`,
//! `--mib` MiB decoded each, shape `[mib * 256, 1024]`, chunks `256 x 256`,
//! deflate level 4.
//! * `<dir>/contiguous.h5`: the same datasets, contiguous.
//!
//! Modes, for each layout and each thread count `T` (strong scaling: the total
//! work per repetition is fixed, split among the threads):
//!
//! * `distinct`: every dataset is read in full once; thread `t` reads datasets
//! `t, t + T, t + 2T, ...`.
//! * `same`: all threads read `d00`, `--slabs` random `--slab` x `--slab`
//! hyperslabs in total (slab `j` goes to thread `j % T`). The offsets come
//! from a splitmix64 stream seeded with `--seed`, identical in the h5py
//! script.
//!
//! One `File` per layout per repetition is shared by all threads (opened
//! fresh each repetition, so no chunk cache carries over). Page cache:
//! `warm` (default) reads every file once before timing; `--cold` evicts the
//! files from the page cache with `posix_fadvise(POSIX_FADV_DONTNEED)` before
//! every repetition (no root needed; it only evicts clean, unmapped pages, so
//! it is best effort — the JSON says which was used).
//!
//! Decode inside one read is itself parallel when clawhdf5-format's `parallel`
//! feature is on (it is in this binary, via clawhdf5-agent). `--decode-threads
//! N` sizes that rayon pool; `--decode-threads 1` measures the API's own
//! thread scaling, comparable with h5py where each call decodes on the
//! calling thread.
//!
//! ```text
//! cargo run --release -p clawhdf5-bench --bin concurrent_read -- \
//! --dir /data/concurrent-read --json clawhdf5.json
//! cargo run --release -p clawhdf5-bench --bin concurrent_read -- \
//! --dir /tmp/cr --datasets 4 --mib 1 --threads 1,2 --slabs 16 --reps 1 # smoke
//! ```
use std::path::{Path, PathBuf};
use std::sync::Barrier;
use std::time::Instant;
use clawhdf5::{File, FileBuilder, Selection};
use serde::{Deserialize, Serialize};
const COLS: u64 = 1024;
const ROWS_PER_MIB: u64 = 256; // 256 rows x 1024 cols x 4 bytes = 1 MiB
const CHUNK: u64 = 256;
const DEFLATE_LEVEL: u32 = 4;
const LAYOUTS: [&str; 2] = ["deflate", "contiguous"];
const MANIFEST_VERSION: u32 = 1;
/// splitmix64 — shared with the h5py script, which must produce the same
/// stream (both the data and the hyperslab offsets depend on it).
fn splitmix64(state: &mut u64) -> u64 {
*state = state.wrapping_add(0x9E37_79B9_7F4A_7C15);
let mut z = *state;
z = (z ^ (z >> 30)).wrapping_mul(0xBF58_476D_1CE4_E5B9);
z = (z ^ (z >> 27)).wrapping_mul(0x94D0_49BB_1331_11EB);
z ^ (z >> 31)
}
/// Element `i` (row-major) of dataset `k`: a slowly varying integer part plus
/// 8 bits of noise, so deflate has real work to do (about 3.1x) and every value
/// is exact in `f32` (< 2^15 with 8 fraction bits), which lets both harnesses
/// check what they read against this formula.
fn value(k: u64, i: u64) -> f32 {
let mut s = i ^ (k << 40);
let noise = splitmix64(&mut s) & 0xff;
(((i >> 6) % 16384) + k) as f32 + noise as f32 / 256.0
}
#[derive(Serialize, Deserialize, PartialEq, Debug, Clone)]
struct Manifest {
version: u32,
datasets: u64,
rows: u64,
cols: u64,
chunk: [u64; 2],
deflate_level: u32,
files: Vec<(String, String)>, // (layout, file name)
writer: String,
}
fn manifest_for(datasets: u64, mib: u64) -> Manifest {
Manifest {
version: MANIFEST_VERSION,
datasets,
rows: mib * ROWS_PER_MIB,
cols: COLS,
chunk: [CHUNK, CHUNK],
deflate_level: DEFLATE_LEVEL,
files: LAYOUTS
.iter()
.map(|l| (l.to_string(), format!("{l}.h5")))
.collect(),
writer: format!("clawhdf5 {}", env!("CARGO_PKG_VERSION")),
}
}
fn dataset_values(k: u64, n: u64) -> Vec<f32> {
(0..n).map(|i| value(k, i)).collect()
}
/// Write the files unless `dir` already holds ones matching `want`.
fn ensure_files(dir: &Path, want: &Manifest) -> std::io::Result<bool> {
let manifest_path = dir.join("manifest.json");
if let Ok(text) = std::fs::read_to_string(&manifest_path)
&& let Ok(have) = serde_json::from_str::<Manifest>(&text)
&& have.version == want.version
&& have.datasets == want.datasets
&& have.rows == want.rows
&& have.cols == want.cols
&& have.chunk == want.chunk
&& have.deflate_level == want.deflate_level
&& have.files == want.files
&& want.files.iter().all(|(_, f)| dir.join(f).exists())
{
return Ok(false);
}
std::fs::create_dir_all(dir)?;
// A stale manifest must not survive a half-written regeneration.
let _ = std::fs::remove_file(&manifest_path);
let n = want.rows * want.cols;
for (layout, file) in &want.files {
// One layout at a time keeps the peak memory to about twice one
// file's decoded size.
let mut b = FileBuilder::new();
for k in 0..want.datasets {
let ds = b.create_dataset(&format!("d{k:02}"));
ds.with_f32_data(&dataset_values(k, n))
.with_shape(&[want.rows, want.cols]);
if layout == "deflate" {
ds.with_chunks(&[CHUNK.min(want.rows), CHUNK])
.with_deflate(DEFLATE_LEVEL);
}
}
b.write(dir.join(file)).map_err(std::io::Error::other)?;
}
std::fs::write(
&manifest_path,
serde_json::to_string_pretty(want).map_err(std::io::Error::other)?,
)?;
Ok(true)
}
fn slab_offsets(seed: u64, count: usize, rows: u64, cols: u64, slab: u64) -> Vec<(u64, u64)> {
let mut s = seed;
(0..count)
.map(|_| {
let r = splitmix64(&mut s) % (rows - slab + 1);
let c = splitmix64(&mut s) % (cols - slab + 1);
(r, c)
})
.collect()
}
/// Warm the page cache by reading every byte of `path`.
fn warm(path: &Path) -> std::io::Result<()> {
let mut f = std::fs::File::open(path)?;
std::io::copy(&mut f, &mut std::io::sink())?;
Ok(())
}
/// Ask the kernel to drop `path`'s pages from the page cache.
fn evict(path: &Path) -> std::io::Result<()> {
use std::os::fd::AsRawFd;
let f = std::fs::File::open(path)?;
// SAFETY: plain syscall on a valid, open file descriptor.
let rc = unsafe { libc::posix_fadvise(f.as_raw_fd(), 0, 0, libc::POSIX_FADV_DONTNEED) };
if rc != 0 {
return Err(std::io::Error::from_raw_os_error(rc));
}
Ok(())
}
#[derive(Serialize)]
struct Row {
layout: String,
mode: String,
threads: usize,
/// Decoded (selected) bytes read per repetition.
bytes: u64,
times_s: Vec<f64>,
median_s: f64,
mb_s: f64,
/// `mb_s / (threads * mb_s at threads = 1)`; null without a 1-thread row.
efficiency: Option<f64>,
}
struct Args {
dir: PathBuf,
datasets: u64,
mib: u64,
threads: Vec<usize>,
reps: usize,
slab: u64,
slabs: usize,
seed: u64,
cold: bool,
decode_threads: usize,
modes: Vec<String>,
layouts: Vec<String>,
json: Option<PathBuf>,
}
const USAGE: &str = "\
usage: concurrent_read [--dir DIR] [--datasets N] [--mib N] [--threads 1,2,4,8,16]
[--reps N] [--slab N] [--slabs N] [--seed N] [--cold]
[--decode-threads N] [--modes distinct,same]
[--layouts deflate,contiguous] [--json FILE]";
fn parse_list<T: std::str::FromStr>(s: &str) -> Result<Vec<T>, String> {
s.split(',')
.map(|x| x.trim().parse().map_err(|_| format!("bad list item {x:?}")))
.collect()
}
fn parse_args() -> Result<Args, String> {
let mut a = Args {
dir: PathBuf::from("concurrent-read-data"),
datasets: 64,
mib: 64,
threads: vec![1, 2, 4, 8, 16],
reps: 3,
slab: 256,
slabs: 1024,
seed: 42,
cold: false,
decode_threads: 0,
modes: vec!["distinct".into(), "same".into()],
layouts: LAYOUTS.iter().map(|s| s.to_string()).collect(),
json: None,
};
let mut it = std::env::args().skip(1);
while let Some(flag) = it.next() {
if flag == "--cold" {
a.cold = true;
continue;
}
if flag == "-h" || flag == "--help" {
return Err(USAGE.into());
}
let v = it.next().ok_or(format!("{flag} needs a value\n{USAGE}"))?;
let num = |v: &str| {
v.parse::<u64>()
.map_err(|_| format!("{flag}: bad number {v:?}"))
};
match flag.as_str() {
"--dir" => a.dir = v.into(),
"--datasets" => a.datasets = num(&v)?,
"--mib" => a.mib = num(&v)?,
"--threads" => a.threads = parse_list(&v)?,
"--reps" => a.reps = num(&v)? as usize,
"--slab" => a.slab = num(&v)?,
"--slabs" => a.slabs = num(&v)? as usize,
"--seed" => a.seed = num(&v)?,
"--decode-threads" => a.decode_threads = num(&v)? as usize,
"--modes" => a.modes = parse_list(&v)?,
"--layouts" => a.layouts = parse_list(&v)?,
"--json" => a.json = Some(v.into()),
_ => return Err(format!("unknown flag {flag}\n{USAGE}")),
}
}
if a.datasets == 0 || a.datasets > 100 {
return Err("--datasets must be 1..=100".into());
}
if a.mib == 0 || a.reps == 0 || a.slabs == 0 || a.threads.contains(&0) {
return Err("--mib, --reps, --slabs and every --threads value must be > 0".into());
}
if a.slab == 0 || a.slab > COLS || a.slab > a.mib * ROWS_PER_MIB {
return Err(format!(
"--slab must be 1..={}",
COLS.min(a.mib * ROWS_PER_MIB)
));
}
for m in &a.modes {
if m != "distinct" && m != "same" {
return Err(format!("unknown mode {m:?}"));
}
}
for l in &a.layouts {
if !LAYOUTS.contains(&l.as_str()) {
return Err(format!("unknown layout {l:?}"));
}
}
Ok(a)
}
/// One timed repetition: `T` threads on one shared `File`. Returns seconds.
fn run_once(
path: &Path,
mode: &str,
threads: usize,
m: &Manifest,
slabs: &[(u64, u64)],
slab: u64,
verify: bool,
) -> f64 {
let file = File::open(path).expect("open");
let barrier = Barrier::new(threads + 1); // + the spawning thread
let n = m.rows * m.cols;
// Each thread times itself from the barrier; the repetition spans the
// earliest start to the latest finish (timing on the spawning thread
// instead undercounts whenever it is scheduled after the workers ran).
let spans: Vec<(Instant, Instant)> = std::thread::scope(|s| {
let handles: Vec<_> = (0..threads)
.map(|t| {
let (file, barrier) = (&file, &barrier);
s.spawn(move || {
barrier.wait();
let start = Instant::now();
match mode {
"distinct" => {
for k in (t as u64..m.datasets).step_by(threads) {
let got = file.dataset(&format!("d{k:02}")).unwrap().read_f32();
let got = got.unwrap();
assert_eq!(got.len() as u64, n);
if verify {
for i in [0, n / 3, n - 1] {
assert_eq!(got[i as usize], value(k, i), "d{k:02}[{i}]");
}
}
std::hint::black_box(got);
}
}
_ => {
let ds = file.dataset("d00").unwrap();
for &(r, c) in slabs.iter().skip(t).step_by(threads) {
let sel = Selection::Hyperslab {
start: vec![r, c],
stride: vec![1, 1],
count: vec![slab, slab],
block: vec![1, 1],
};
let got = ds.read_f32_selection(&sel).unwrap();
assert_eq!(got.len() as u64, slab * slab);
if verify {
let last = (r + slab - 1) * m.cols + c + slab - 1;
assert_eq!(got[0], value(0, r * m.cols + c));
assert_eq!(*got.last().unwrap(), value(0, last));
}
std::hint::black_box(got);
}
}
}
(start, Instant::now())
})
})
.collect();
barrier.wait();
handles.into_iter().map(|h| h.join().unwrap()).collect()
});
let start = spans.iter().map(|s| s.0).min().unwrap();
let end = spans.iter().map(|s| s.1).max().unwrap();
(end - start).as_secs_f64()
}
fn median(v: &[f64]) -> f64 {
let mut s = v.to_vec();
s.sort_by(f64::total_cmp);
s[s.len() / 2]
}
fn hostname() -> String {
std::fs::read_to_string("/proc/sys/kernel/hostname")
.map(|s| s.trim().to_string())
.unwrap_or_else(|_| "unknown".into())
}
fn main() {
let args = match parse_args() {
Ok(a) => a,
Err(e) => {
eprintln!("{e}");
std::process::exit(2);
}
};
if cfg!(debug_assertions) {
eprintln!("warning: debug build — numbers are meaningless. Use --release.");
}
if args.decode_threads > 0 {
rayon::ThreadPoolBuilder::new()
.num_threads(args.decode_threads)
.build_global()
.expect("configure rayon pool");
}
let manifest = manifest_for(args.datasets, args.mib);
let t = Instant::now();
match ensure_files(&args.dir, &manifest) {
Ok(true) => eprintln!(
"generated {} in {:.1} s",
args.dir.display(),
t.elapsed().as_secs_f64()
),
Ok(false) => eprintln!("reusing {}", args.dir.display()),
Err(e) => {
eprintln!("cannot write test files in {}: {e}", args.dir.display());
std::process::exit(1);
}
}
let path_of = |layout: &str| args.dir.join(format!("{layout}.h5"));
let slabs = slab_offsets(
args.seed,
args.slabs,
manifest.rows,
manifest.cols,
args.slab,
);
let dataset_bytes = manifest.rows * manifest.cols * 4;
let mut rows: Vec<Row> = Vec::new();
println!("| layout | mode | threads | MB/s | efficiency | median s |");
println!("|---|---|---:|---:|---:|---:|");
for layout in &args.layouts {
let path = path_of(layout);
// Untimed pass: page cache warm (unless --cold), results checked.
if !args.cold {
warm(&path).expect("warm page cache");
}
for mode in &args.modes {
run_once(&path, mode, 1, &manifest, &slabs, args.slab, true);
let bytes = match mode.as_str() {
"distinct" => dataset_bytes * manifest.datasets,
_ => args.slab * args.slab * 4 * args.slabs as u64,
};
let mut base: Option<f64> = None;
for &threads in &args.threads {
let times: Vec<f64> = (0..args.reps)
.map(|_| {
if args.cold {
evict(&path).expect("posix_fadvise");
}
run_once(&path, mode, threads, &manifest, &slabs, args.slab, false)
})
.collect();
let med = median(&times);
let mb_s = bytes as f64 / (1 << 20) as f64 / med;
if threads == 1 {
base = Some(mb_s);
}
let efficiency = base.map(|b| mb_s / (threads as f64 * b));
println!(
"| {layout} | {mode} | {threads} | {mb_s:.0} | {} | {med:.4} |",
efficiency.map_or("-".into(), |e| format!("{e:.2}"))
);
rows.push(Row {
layout: layout.clone(),
mode: mode.clone(),
threads,
bytes,
times_s: times,
median_s: med,
mb_s,
efficiency,
});
}
}
}
if let Some(out) = &args.json {
let doc = serde_json::json!({
"tool": "clawhdf5",
"version": env!("CARGO_PKG_VERSION"),
"host": hostname(),
"cpus": std::thread::available_parallelism().map_or(0, |n| n.get()),
"unix_time": std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map_or(0, |d| d.as_secs()),
"cache": if args.cold { "cold (posix_fadvise DONTNEED before each repetition)" } else { "warm" },
"decode_threads": rayon::current_num_threads(),
"params": {
"datasets": manifest.datasets,
"mib": args.mib,
"rows": manifest.rows,
"cols": manifest.cols,
"chunk": manifest.chunk,
"deflate_level": manifest.deflate_level,
"slab": args.slab,
"slabs": args.slabs,
"seed": args.seed,
"reps": args.reps,
"dir": args.dir,
},
"results": rows,
});
std::fs::write(out, serde_json::to_string_pretty(&doc).unwrap()).expect("write json");
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn values_are_exact_in_f32() {
for k in [0, 7, 63] {
for i in [0u64, 1, 4095, 1 << 20, (1 << 24) - 1] {
let v = value(k, i);
assert_eq!(v, (v as f64) as f32);
assert!(v < 32768.0);
assert_eq!((v * 256.0).fract(), 0.0);
}
}
}
/// The h5py script hard-codes this vector to check its splitmix64 port.
#[test]
fn splitmix64_reference() {
let mut s = 42;
assert_eq!(splitmix64(&mut s), 0xBDD7_3226_2FEB_6E95);
}
}
crates/clawhdf5-bench/src/bin/consolidation_efficiency.rs
+4 -3
View File
@@ -396,7 +396,7 @@ fn run_memory_reduction_benchmark() {
println!();
println!(
"{:>8} {:>10} {:>10} {:>10} {:>12}",
"Initial", "Remaining", "Eviction%", "Signal OK?", "BM25 Speedup"
"Initial", "Remaining", "Eviction%", "Signal OK?", "Records ÷"
);
println!("{}", "-".repeat(58));
@@ -440,7 +440,8 @@ fn run_memory_reduction_benchmark() {
// Check all signal records survived
let signal_survived = signal_ids.iter().all(|&id| engine.get_by_id(id).is_some());
// Rough speedup: BM25 scales roughly linearly with record count
// How many times fewer records there are. Not a measured speedup —
// Part 1 measures search latency before and after.
let speedup = before_count as f64 / after_count.max(1) as f64;
println!(
@@ -480,7 +481,7 @@ fn main() {
println!(" 3. Reducing search latency proportional to record reduction");
println!();
println!(
"Cycle time scales sub-linearly: 100 records ~microseconds, 100K records ~tens of ms."
"Cycle time grows a little faster than linearly: 100 records ~microseconds, 100K records ~tens of ms."
);
println!("Signal records with Correction source + high access_count survive eviction.");
}
crates/clawhdf5-bench/src/bin/footprint_bench.rs
+19 -3
View File
@@ -11,12 +11,14 @@
//!
//! Configuration matrix:
//! - Text lengths: short (50 chars), medium (200 chars), long (1000 chars)
//! - Embedding: 384-dim f32 (1536 bytes raw per record)
//! - Embedding: 384-dim, stored as float16 (the default for new stores) or
//! f32 with `--f32`; "raw" bytes are counted as f32 input either way
//! - WAL: enabled and disabled
//!
//! # Usage
//! ```
//! cargo run --release --bin footprint_bench
//! cargo run --release --bin footprint_bench # float16 stores
//! cargo run --release --bin footprint_bench -- --f32 # f32 stores
//! ```
use std::time::Instant;
@@ -24,6 +26,9 @@ use std::time::Instant;
use clawhdf5_agent::{AgentMemory, HDF5Memory, MemoryConfig, MemoryEntry};
use tempfile::TempDir;
/// `--f32`: build f32 stores instead of the library's float16 default.
static F32: std::sync::atomic::AtomicBool = std::sync::atomic::AtomicBool::new(false);
const EMBEDDING_DIM: usize = 384;
// Raw bytes per record: 384 f32 embeddings + median text + overhead
@@ -152,6 +157,9 @@ fn measure_footprint(
config.compression = compression;
config.compression_level = if compression { 6 } else { 0 };
config.compact_threshold = 0.0;
if F32.load(std::sync::atomic::Ordering::Relaxed) {
config.float16 = false;
}
let mut memory = HDF5Memory::create(config).expect("HDF5Memory::create failed");
@@ -241,11 +249,19 @@ fn fmt_n(n: usize) -> String {
// ---------------------------------------------------------------------------
fn main() {
if std::env::args().skip(1).any(|a| a == "--f32") {
F32.store(true, std::sync::atomic::Ordering::Relaxed);
}
let stored = if F32.load(std::sync::atomic::Ordering::Relaxed) {
"f32 (1,536 bytes per record)"
} else {
"float16 (768 bytes per record; the default for new stores)"
};
println!("=================================================================");
println!(" ClawhDF5 Memory Footprint Benchmark");
println!("=================================================================");
println!();
println!("Embedding: 384-dim f32 = 1,536 bytes raw per record");
println!("Embedding: 384-dim, stored as {stored}; raw input counted as f32");
println!("Text lengths: short=50 chars, medium=200 chars, long=1000 chars");
println!();
crates/clawhdf5-bench/src/bin/longmemeval_bench.rs
+376 -44
View File
@@ -55,44 +55,155 @@ use std::time::{Duration, Instant};
#[path = "longmemeval_bench/embedder.rs"]
mod embedder;
use clawhdf5_agent::{AgentMemory, HDF5Memory, MemoryConfig, MemoryEntry};
use clawhdf5_agent::bm25::TokenFilter;
use clawhdf5_agent::hybrid::Fusion;
use clawhdf5_agent::reranker::{ReRankConfig, RerankInput, rerank};
use clawhdf5_agent::{AgentMemory, HDF5Memory, MemoryConfig, MemoryEntry, SearchResult};
use serde::Deserialize;
use tempfile::TempDir;
const EMBEDDING_DIM: usize = 384;
/// `--float16`: build every per-question store with `MemoryConfig::float16`,
/// so embeddings are rounded to half precision as they are saved — exactly
/// what such a store searches over.
static FLOAT16: std::sync::atomic::AtomicBool = std::sync::atomic::AtomicBool::new(false);
/// A mode's fusion, as one short string for the reports.
fn describe(mode: Mode) -> String {
let fusion = match mode.fusion {
Fusion::Weighted { vector, keyword } => format!("vector_{vector:.1}_keyword_{keyword:.1}"),
Fusion::Rrf { k } => format!("rrf_k{k:.0}"),
};
let tokens = match mode.tokens {
TokenFilter::Plain => fusion,
TokenFilter::Stemmed => format!("{fusion}_stemmed"),
};
match mode.rerank {
None => tokens,
Some(cfg) if cfg.relevance_weight == 0.0 => format!("{tokens}_rerank_metadata"),
Some(cfg) => format!(
"{tokens}_rerank_blended_hl{:.0}d",
cfg.temporal_half_life_secs / 86_400.0
),
}
}
/// A retrieval configuration: how much of the score comes from each stage.
#[derive(Clone, Copy)]
struct Mode {
label: &'static str,
vector_weight: f32,
keyword_weight: f32,
/// How the two retrieval stages are combined into one ranking.
fusion: Fusion,
/// How keyword tokens are normalised before indexing and querying.
tokens: TokenFilter,
/// Re-rank the retrieved candidates with recency and friends, relative to
/// the question's own date.
rerank: Option<ReRankConfig>,
}
impl Mode {
const fn weighted(label: &'static str, vector: f32, keyword: f32) -> Self {
Self {
label,
fusion: Fusion::Weighted { vector, keyword },
tokens: TokenFilter::Plain,
rerank: None,
}
}
#[cfg_attr(not(feature = "embeddings"), allow(dead_code))]
fn reranked(mut self, label: &'static str, rerank: ReRankConfig) -> Self {
self.label = label;
self.rerank = Some(rerank);
self
}
const fn stemmed(mut self, label: &'static str) -> Self {
self.label = label;
self.tokens = TokenFilter::Stemmed;
self
}
}
/// The only mode available without real embeddings. Passing zero vectors with
/// `vector_weight = 0.0` is what made the vector stage inert.
const BM25_ONLY: Mode = Mode {
label: "BM25 only (vector stage inert)",
vector_weight: 0.0,
keyword_weight: 1.0,
};
const BM25_ONLY: Mode = Mode::weighted("BM25 only (vector stage inert)", 0.0, 1.0);
#[cfg(feature = "embeddings")]
const VECTOR_ONLY: Mode = Mode {
label: "Vector only (MiniLM + HNSW)",
vector_weight: 1.0,
keyword_weight: 0.0,
};
const VECTOR_ONLY: Mode = Mode::weighted("Vector only (MiniLM + HNSW)", 1.0, 0.0);
/// Tuned by `--sweep` over the full haystack. The former 0.7/0.3 was a
/// documented default that had never been searched, and the sweep found it
/// strictly dominated: 0.4/0.6 is better on Hit@1, Hit@5, Hit@10 and MRR at
/// both granularities.
#[cfg(feature = "embeddings")]
const HYBRID: Mode = Mode {
label: "Hybrid (0.4 vector / 0.6 BM25, tuned)",
vector_weight: 0.4,
keyword_weight: 0.6,
const HYBRID: Mode = Mode::weighted("Hybrid (0.4 vector / 0.6 BM25, tuned)", 0.4, 0.6);
/// Reciprocal rank fusion, the documented alternative to the weighted sum.
/// It ignores score magnitudes, so there is nothing to tune — which is the
/// claim being tested.
#[cfg(feature = "embeddings")]
const RRF: Mode = Mode {
label: "Hybrid (reciprocal rank fusion, k=60)",
fusion: Fusion::Rrf { k: 60.0 },
tokens: TokenFilter::Plain,
rerank: None,
};
/// The same two configurations with stemmed keyword tokens, so the tokenizer's
/// effect is isolated from everything else.
const BM25_STEMMED: Mode = BM25_ONLY.stemmed("BM25 only, stemmed tokens");
/// Re-ranking as it behaved before `relevance` was an input: the combined
/// score was recency + authority + activation only, so the retriever's own
/// ordering was discarded.
#[cfg(feature = "embeddings")]
fn hybrid_rerank_metadata_only() -> Mode {
HYBRID.reranked(
"Hybrid + rerank (metadata only, pre-fix)",
ReRankConfig {
relevance_weight: 0.0,
..ReRankConfig::default()
},
)
}
/// Re-ranking as it behaves now: relevance leads, recency nudges.
#[cfg(feature = "embeddings")]
fn hybrid_rerank_blended() -> Mode {
HYBRID.reranked(
"Hybrid + rerank (relevance + recency)",
ReRankConfig::default(),
)
}
/// The same blend at several half-lives. Decay is `2^(-age / half_life)`, so a
/// half-life far shorter than the gaps between memories sends every score to
/// zero and the signal vanishes; far longer and everything scores ~1 and it
/// vanishes the other way. The right value tracks how far apart the memories
/// actually are.
#[cfg(feature = "embeddings")]
fn hybrid_rerank_half_lives() -> Vec<Mode> {
[
("1 day", 86_400.0),
("7 days", 7.0 * 86_400.0),
("30 days", 30.0 * 86_400.0),
("90 days", 90.0 * 86_400.0),
]
.into_iter()
.map(|(label, half_life)| {
HYBRID.reranked(
Box::leak(format!("Hybrid + rerank, half-life {label}").into_boxed_str()),
ReRankConfig {
temporal_half_life_secs: half_life,
..ReRankConfig::default()
},
)
})
.collect()
}
#[cfg(feature = "embeddings")]
const HYBRID_STEMMED: Mode = HYBRID.stemmed("Hybrid 0.4/0.6, stemmed tokens");
/// Every 0.1 step of vector weight, keyword weight taking the remainder.
///
/// Labels are leaked to `&'static str` because `Mode::label` is a `&'static
@@ -104,11 +215,11 @@ fn sweep_modes() -> Vec<Mode> {
(0..=10)
.map(|i| {
let v = i as f32 / 10.0;
Mode {
label: Box::leak(format!("sweep v={v:.1} / k={:.1}", 1.0 - v).into_boxed_str()),
vector_weight: v,
keyword_weight: 1.0 - v,
}
Mode::weighted(
Box::leak(format!("sweep v={v:.1} / k={:.1}", 1.0 - v).into_boxed_str()),
v,
1.0 - v,
)
})
.collect()
}
@@ -181,6 +292,37 @@ struct Question {
haystack_session_ids: Vec<String>,
haystack_sessions: Vec<Vec<Turn>>,
answer_session_ids: Vec<String>,
/// One timestamp per haystack session, e.g. "2023/05/25 (Thu) 20:21".
#[serde(default)]
haystack_dates: Vec<String>,
}
/// Seconds since the epoch for a LongMemEval session date, which looks like
/// `2023/05/25 (Thu) 20:21`. Sessions are stored in chronological order, so a
/// date that cannot be parsed falls back to its position — order is preserved
/// even if the interval is not.
fn session_time(date: &str, position: usize) -> f64 {
let stamp = |y: i64, mo: i64, d: i64, h: i64, mi: i64| -> f64 {
// Days since 1970-01-01 via the civil-from-days algorithm.
let (y, mo) = if mo <= 2 { (y - 1, mo + 12) } else { (y, mo) };
let era = y.div_euclid(400);
let yoe = y - era * 400;
let doy = (153 * (mo - 3) + 2) / 5 + d - 1;
let doe = yoe * 365 + yoe / 4 - yoe / 100 + doy;
let days = era * 146_097 + doe - 719_468;
(days * 86_400 + h * 3_600 + mi * 60) as f64
};
let parse = || -> Option<f64> {
let (ymd, rest) = date.split_once(' ')?;
let mut ymd = ymd.split('/');
let y = ymd.next()?.parse().ok()?;
let mo = ymd.next()?.parse().ok()?;
let d = ymd.next()?.parse().ok()?;
let hm = rest.rsplit(' ').next()?;
let (h, mi) = hm.split_once(':')?;
Some(stamp(y, mo, d, h.parse().ok()?, mi.parse().ok()?))
};
parse().unwrap_or(1_000_000.0 + position as f64 * 86_400.0)
}
// ---------------------------------------------------------------------------
@@ -199,11 +341,21 @@ struct Metrics {
rr_turn: f64,
abstention_correct: u32,
abstention_total: u32,
/// Questions where the newest gold session outranked the older ones, out
/// of those with more than one gold session and at least one retrieved.
newest_gold_first: u32,
newest_gold_total: u32,
latency_ns: Vec<u64>,
count: u32,
}
impl Metrics {
/// `None` when no question in this bucket had multiple gold sessions.
fn newest_gold_first_pct(&self) -> Option<f64> {
(self.newest_gold_total > 0)
.then(|| self.newest_gold_first as f64 / self.newest_gold_total as f64 * 100.0)
}
fn hit1_session_pct(&self) -> f64 {
self.hit1_session as f64 / self.count.max(1) as f64 * 100.0
}
@@ -261,6 +413,16 @@ struct EvalResult {
hit5_turn: bool,
hit10_turn: bool,
rr_turn: Option<f64>,
/// For a question whose evidence spans several dated sessions (a
/// `knowledge-update`, where an earlier fact is superseded by a later
/// one): did the *newest* gold session outrank every older gold session
/// that was returned? `None` when the question has one gold session, or
/// when none were retrieved, so there is nothing to discriminate.
///
/// Plain recall cannot see this. LongMemEval labels *both* the stale and
/// the updated session as gold, so returning either counts as a hit — yet
/// only one of them answers the question correctly.
newest_gold_first: Option<bool>,
latency: Duration,
}
@@ -274,21 +436,29 @@ fn evaluate_question(
let mut config = MemoryConfig::new(dir.path().join("lme.h5"), "lme-bench", EMBEDDING_DIM);
config.wal_enabled = false;
config.compact_threshold = 0.0;
config.float16 = FLOAT16.load(std::sync::atomic::Ordering::Relaxed);
let mut memory = HDF5Memory::create(config).expect("failed to create HDF5Memory");
memory.set_token_filter(mode.tokens);
// Build MemoryEntry list from all haystack sessions
let mut entries: Vec<MemoryEntry> = Vec::new();
let mut turn_has_answer: Vec<bool> = Vec::new();
let mut ts = 1_000_000.0f64;
for (sess_idx, session) in q.haystack_sessions.iter().enumerate() {
let sess_id = q
.haystack_session_ids
.get(sess_idx)
.map(String::as_str)
.unwrap_or("unknown");
for turn in session {
// Real session dates, not a synthetic counter: anything that decays
// with age needs true intervals, not just the right order.
let session_start = q
.haystack_dates
.get(sess_idx)
.map_or(sess_idx as f64 * 86_400.0, |d| session_time(d, sess_idx));
for (turn_idx, turn) in session.iter().enumerate() {
// Spread a session's turns over the minutes following its start.
let ts = session_start + turn_idx as f64 * 60.0;
entries.push(MemoryEntry {
chunk: turn.content.clone(),
embedding: embedding_for(embeddings, &turn.content),
@@ -302,7 +472,6 @@ fn evaluate_question(
},
});
turn_has_answer.push(turn.has_answer);
ts += 1.0;
}
}
@@ -319,17 +488,87 @@ fn evaluate_question(
// Set of session IDs that contain the answer
let answer_sess_set: HashSet<&str> = q.answer_session_ids.iter().map(String::as_str).collect();
// When each gold session was recorded, so "newest" is by date rather than
// by position (the two agree in this dataset, but the metric should not
// depend on that).
let gold_times: HashMap<&str, f64> = q
.haystack_session_ids
.iter()
.enumerate()
.filter(|(_, sid)| answer_sess_set.contains(sid.as_str()))
.map(|(i, sid)| {
let t = q
.haystack_dates
.get(i)
.map_or(i as f64 * 86_400.0, |d| session_time(d, i));
(sid.as_str(), t)
})
.collect();
let query_emb = embedding_for(embeddings, &q.question);
let t0 = Instant::now();
let results = memory.hybrid_search(
&query_emb,
&q.question,
mode.vector_weight,
mode.keyword_weight,
top_k,
);
// Re-ranking only reorders; it needs a candidate pool larger than `top_k`
// to have anything to promote.
let pool = if mode.rerank.is_some() {
top_k * 4
} else {
top_k
};
let mut results = memory.hybrid_search_with(&query_emb, &q.question, mode.fusion, pool);
if let Some(config) = mode.rerank {
// "Now" is the moment the question was asked, so decay measures how
// stale each memory was at that point.
let now = session_time(&q.question_date, q.haystack_sessions.len());
let inputs: Vec<RerankInput> = results
.iter()
.map(|r| RerankInput {
index: r.index,
timestamp: r.timestamp,
source_channel: r.source_channel.clone(),
raw_activation: r.activation,
relevance: r.score,
})
.collect();
let order: Vec<usize> = rerank(&inputs, &config, now)
.into_iter()
.map(|r| r.index)
.collect();
let by_index: HashMap<usize, SearchResult> =
results.into_iter().map(|r| (r.index, r)).collect();
results = order
.into_iter()
.filter_map(|i| by_index.get(&i).cloned())
.collect();
}
results.truncate(top_k);
let latency = t0.elapsed();
// Rank of the best-placed result from each gold session.
let mut first_rank: HashMap<&str, usize> = HashMap::new();
for (rank, result) in results.iter().enumerate() {
let sid = memory.cache.session_ids[result.index].as_str();
if let Some((gold_sid, _)) = gold_times.get_key_value(sid) {
first_rank.entry(gold_sid).or_insert(rank);
}
}
let newest_gold_first = if gold_times.len() < 2 || first_rank.is_empty() {
None
} else {
// The newest gold session must be retrieved, and no older gold session
// may outrank it.
let newest = gold_times
.iter()
.max_by(|a, b| a.1.total_cmp(b.1))
.map(|(sid, _)| *sid)
.expect("at least two gold sessions");
Some(match first_rank.get(newest) {
Some(&newest_rank) => first_rank
.iter()
.all(|(sid, &rank)| *sid == newest || rank > newest_rank),
None => false,
})
};
// Session-level recall
let mut hit1_session = false;
let mut hit5_session = false;
@@ -384,6 +623,7 @@ fn evaluate_question(
hit5_turn,
hit10_turn,
rr_turn,
newest_gold_first,
latency,
}
}
@@ -472,10 +712,7 @@ fn print_report(
println!(" LongMemEval Benchmark — {}", mode.label);
println!("=================================================================");
println!();
println!(
"Mode: vector_weight={:.1} / keyword_weight={:.1}",
mode.vector_weight, mode.keyword_weight
);
println!("Mode: {}", describe(mode));
println!();
println!("Scoring target: RETRIEVAL RECALL (did the gold memory land in top-k).");
println!(" No answer is generated or scored. This is NOT the official");
@@ -538,6 +775,24 @@ fn print_report(
);
println!();
if let Some(pct) = overall.newest_gold_first_pct() {
println!(
"## Recency Discrimination (n={})",
overall.newest_gold_total
);
println!(
" Newest gold session ranked first: {}/{} ({pct:.1}%)",
overall.newest_gold_first, overall.newest_gold_total
);
println!(
" Questions whose evidence spans several dated sessions — a fact and\n \
its later correction. Both sessions are labelled gold, so recall\n \
scores either as a hit; this asks whether the *current* one came\n \
first. A retriever with no sense of time scores near chance."
);
println!();
}
if overall.abstention_total > 0 {
println!("## Abstention Accuracy");
println!(
@@ -602,10 +857,7 @@ fn print_report(
println!("```json");
println!("{{");
println!(" \"benchmark\": \"longmemeval\",");
println!(
" \"mode\": \"vector_{:.1}_keyword_{:.1}\",",
mode.vector_weight, mode.keyword_weight
);
println!(" \"mode\": \"{}\",", describe(mode));
println!(" \"dataset_variant\": \"{}\",", profile.variant());
println!(" \"scoring_target\": \"retrieval_recall\",");
println!(" \"k\": 10,");
@@ -654,6 +906,14 @@ fn print_report(
} else {
println!(" \"abstention_accuracy\": null,");
}
match overall.newest_gold_first_pct() {
Some(pct) => println!(
" \"newest_gold_first\": {:.4}, \"newest_gold_n\": {},",
pct / 100.0,
overall.newest_gold_total
),
None => println!(" \"newest_gold_first\": null,"),
}
println!(" \"latency_us\": {{");
println!(
" \"avg\": {:.1}, \"p50\": {:.1}, \"p95\": {:.1}, \"p99\": {:.1}",
@@ -676,6 +936,8 @@ fn main() {
let mut limit: Option<usize> = None;
let mut weights_dir: Option<String> = None;
let mut sweep = false;
#[cfg_attr(not(feature = "embeddings"), allow(unused_mut, unused_variables))]
let mut rerank_sweep = false;
let mut args = std::env::args().skip(1);
while let Some(arg) = args.next() {
match arg.as_str() {
@@ -684,6 +946,20 @@ fn main() {
limit = Some(v.parse().expect("--limit must be a positive integer"));
}
"--sweep" => sweep = true,
"--float16" => {
FLOAT16.store(true, std::sync::atomic::Ordering::Relaxed);
eprintln!("Stores use MemoryConfig::float16 (half-precision embeddings)");
}
"--rerank-sweep" => {
// Re-ranking needs the vector stage to have candidates worth
// reordering, so this is an embeddings-only comparison.
#[cfg(feature = "embeddings")]
{
rerank_sweep = true;
}
#[cfg(not(feature = "embeddings"))]
eprintln!("warning: --rerank-sweep needs --features embeddings; ignoring");
}
"--embeddings" => {
weights_dir = Some(args.next().expect("--embeddings needs a directory"));
}
@@ -702,6 +978,12 @@ fn main() {
BM25-only, vector-only, and hybrid separately. Requires\n\
--features embeddings; without it the vector stage is\n\
inert and only the BM25 row is produced.\n\
--rerank-sweep\n\
compare re-ranking off, metadata-only (the old\n\
behaviour) and blended at several half-lives.\n\
--float16\n\
build each store with MemoryConfig::float16, to\n\
compare retrieval on half-precision embeddings.\n\
--sweep instead of the three named modes, sweep vector_weight\n\
from 0.0 to 1.0 in 0.1 steps. The 0.7/0.3 default was\n\
never searched; this is what searches it."
@@ -767,19 +1049,34 @@ fn main() {
{
if sweep {
sweep_modes()
} else if rerank_sweep {
let mut modes = vec![HYBRID, hybrid_rerank_metadata_only()];
modes.extend(hybrid_rerank_half_lives());
modes
} else {
vec![BM25_ONLY, VECTOR_ONLY, HYBRID]
vec![
BM25_ONLY,
VECTOR_ONLY,
HYBRID,
RRF,
BM25_STEMMED,
HYBRID_STEMMED,
hybrid_rerank_metadata_only(),
hybrid_rerank_blended(),
]
}
}
#[cfg(not(feature = "embeddings"))]
{
vec![BM25_ONLY]
vec![BM25_ONLY, BM25_STEMMED]
}
} else {
if sweep {
eprintln!("warning: --sweep needs --embeddings; running BM25 only");
}
vec![BM25_ONLY]
// Stemming is a property of the keyword stage, so it can be compared
// without a model.
vec![BM25_ONLY, BM25_STEMMED]
};
for (mode_idx, mode) in modes.iter().enumerate() {
@@ -882,6 +1179,14 @@ fn run_mode(
entry.rr_turn += rr;
overall.rr_turn += rr;
}
if let Some(newest_first) = result.newest_gold_first {
entry.newest_gold_total += 1;
overall.newest_gold_total += 1;
if newest_first {
entry.newest_gold_first += 1;
overall.newest_gold_first += 1;
}
}
let ns = result.latency.as_nanos() as u64;
entry.latency_ns.push(ns);
@@ -893,3 +1198,30 @@ fn run_mode(
eprintln!();
print_report(&overall, &by_type, profile, mode);
}
#[cfg(test)]
mod tests {
use super::session_time;
#[test]
fn session_dates_parse_to_the_right_instant() {
// Reference values from Python's datetime, UTC.
for (date, expected) in [
("2023/05/25 (Thu) 20:21", 1_685_046_060.0),
("1970/01/01 (Thu) 00:00", 0.0),
("2000/02/29 (Tue) 12:00", 951_825_600.0),
("2023/12/31 (Sun) 23:59", 1_704_067_140.0),
("2024/03/01 (Fri) 00:00", 1_709_251_200.0),
] {
assert_eq!(session_time(date, 0), expected, "{date}");
}
}
#[test]
fn unparseable_dates_fall_back_to_position_order() {
let a = session_time("not a date", 0);
let b = session_time("", 1);
let c = session_time("2023/13/99 (???) 99:99", 2);
assert!(a < b && b < c, "fallback must preserve session order");
}
}
crates/clawhdf5-bench/src/bin/read_harness.rs
+176
View File
@@ -0,0 +1,176 @@
//! HDF5 read-path measurement harness: full reads vs. hyperslab selections on
//! a chunked 2-D dataset, compressed and uncompressed, plus a contiguous one.
//!
//! The question it answers for every read-path change: does the cost of a
//! selection scale with the *selection*, or with the whole dataset?
//!
//! ```text
//! cargo run --release -p clawhdf5-bench --bin read_harness
//! cargo run --release -p clawhdf5-bench --bin read_harness -- --large # 512 MB
//! ```
use std::time::{Duration, Instant};
use clawhdf5::{File, FileBuilder};
use clawhdf5_format::selection::Selection;
const CHUNK: u64 = 256;
struct Layout {
name: &'static str,
chunked: bool,
deflate: bool,
}
const LAYOUTS: [Layout; 3] = [
Layout {
name: "chunked + deflate",
chunked: true,
deflate: true,
},
Layout {
name: "chunked",
chunked: true,
deflate: false,
},
Layout {
name: "contiguous",
chunked: false,
deflate: false,
},
];
/// Smooth-ish, compressible data whose value encodes its position, so a read
/// can be verified exactly.
fn value(row: u64, col: u64) -> f64 {
(row * 100_003 + col) as f64 * 0.5
}
fn write_file(path: &std::path::Path, rows: u64, cols: u64) {
let data: Vec<f64> = (0..rows)
.flat_map(|r| (0..cols).map(move |c| value(r, c)))
.collect();
let mut builder = FileBuilder::new();
for (i, layout) in LAYOUTS.iter().enumerate() {
let ds = builder.create_dataset(&format!("d{i}"));
ds.with_f64_data(&data).with_shape(&[rows, cols]);
if layout.chunked {
ds.with_chunks(&[CHUNK, CHUNK]);
}
if layout.deflate {
ds.with_deflate(4);
}
}
builder.write(path).unwrap();
}
fn median(mut samples: Vec<Duration>) -> Duration {
samples.sort();
samples[samples.len() / 2]
}
fn time<T>(reps: usize, mut f: impl FnMut() -> T) -> Duration {
median(
(0..reps)
.map(|_| {
let t = Instant::now();
std::hint::black_box(f());
t.elapsed()
})
.collect(),
)
}
fn slab(start: [u64; 2], count: [u64; 2]) -> Selection {
Selection::Hyperslab {
start: start.to_vec(),
stride: vec![1, 1],
count: count.to_vec(),
block: vec![1, 1],
}
}
fn main() {
let large = std::env::args().any(|a| a == "--large");
let (rows, cols) = if large { (8192, 8192) } else { (4096, 2048) };
let total_mb = (rows * cols * 8) as f64 / (1 << 20) as f64;
if cfg!(debug_assertions) {
eprintln!("warning: debug build — numbers are meaningless. Use --release.");
}
let dir = tempfile::TempDir::new().unwrap();
let path = dir.path().join("read_harness.h5");
write_file(&path, rows, cols);
let file_mb = std::fs::metadata(&path).unwrap().len() as f64 / (1 << 20) as f64;
println!("## Read harness");
println!(
"\n{rows} x {cols} f64 ({total_mb:.0} MB per dataset), chunks {CHUNK} x {CHUNK}, file {file_mb:.0} MB\n"
);
// (label, selection, elements selected)
let selections: Vec<(&str, Selection, u64)> = vec![
(
"64 x 64 window (1 chunk)",
slab([300, 300], [64, 64]),
64 * 64,
),
(
"512 x 512 window (4-9 chunks)",
slab([1000, 700], [512, 512]),
512 * 512,
),
("one row", slab([rows / 2, 0], [1, cols]), cols),
("one column", slab([0, cols / 2], [rows, 1]), rows),
];
println!("| layout | read | selected | time ms | MB/s of selection | vs full read |");
println!("|---|---|---:|---:|---:|---:|");
for (i, layout) in LAYOUTS.iter().enumerate() {
// Fresh handle per layout so one dataset's cached chunks don't help
// (or evict) another's.
let file = File::open(&path).unwrap();
let ds = file.dataset(&format!("d{i}")).unwrap();
let full_cold = time(1, || ds.read_f64().unwrap());
let full = time(3, || ds.read_f64().unwrap());
println!(
"| {} | full (first) | {total_mb:.0} MB | {:.1} | {:.0} | |",
layout.name,
full_cold.as_secs_f64() * 1e3,
total_mb / full_cold.as_secs_f64()
);
println!(
"| {} | full (repeat) | {total_mb:.0} MB | {:.1} | {:.0} | 1.00x |",
layout.name,
full.as_secs_f64() * 1e3,
total_mb / full.as_secs_f64()
);
for (label, selection, elements) in &selections {
// A fresh handle again: measure the selection on its own, not
// served from chunks the full read just cached.
let file = File::open(&path).unwrap();
let ds = file.dataset(&format!("d{i}")).unwrap();
let got = ds.read_f64_selection(selection).unwrap();
assert_eq!(got.len() as u64, *elements, "{label}");
if let Selection::Hyperslab { start, .. } = selection {
assert_eq!(got[0], value(start[0], start[1]), "{label}: wrong data");
}
let took = time(5, || {
let file = File::open(&path).unwrap();
let ds = file.dataset(&format!("d{i}")).unwrap();
ds.read_f64_selection(selection).unwrap()
});
let mb = (*elements * 8) as f64 / (1 << 20) as f64;
println!(
"| {} | {label} | {:.2} MB | {:.2} | {:.0} | {:.3}x |",
layout.name,
mb,
took.as_secs_f64() * 1e3,
mb / took.as_secs_f64(),
took.as_secs_f64() / full_cold.as_secs_f64()
);
}
}
}
crates/clawhdf5-bench/src/bin/search_harness.rs
+647 -5
View File
@@ -19,12 +19,15 @@
//! cargo run --release -p clawhdf5-bench --bin search_harness -- --full # + 100K
//! cargo run --release -p clawhdf5-bench --bin search_harness -- --json out.json
//! cargo run --release -p clawhdf5-bench --bin search_harness -- --ann-only --uniform
//! cargo run --release -p clawhdf5-bench --bin search_harness -- --float16-study --full
//! cargo run --release -p clawhdf5-bench --bin search_harness -- --options-study --full
//! cargo run --release -p clawhdf5-bench --bin search_harness -- --signing-study --full
//! ```
use std::time::{Duration, Instant};
use clawhdf5_agent::{AgentMemory, HDF5Memory, MemoryConfig, MemoryEntry};
use clawhdf5_ann::{DistanceMetric, HnswIndex};
use clawhdf5_ann::{DistanceMetric, HnswIndex, Storage};
const DIM: usize = 384;
const K: usize = 10;
@@ -84,6 +87,25 @@ struct Dataset {
/// that appears only on clustered data points at graph connectivity.
static UNIFORM: std::sync::atomic::AtomicBool = std::sync::atomic::AtomicBool::new(false);
/// `--int8`: build the HNSW index over int8-quantised vectors (a quarter of
/// the memory) instead of f32, to price the recall it costs.
static INT8: std::sync::atomic::AtomicBool = std::sync::atomic::AtomicBool::new(false);
/// `--f16-first`: in `--float16-study`, run the float16 store first.
static F16_FIRST: std::sync::atomic::AtomicBool = std::sync::atomic::AtomicBool::new(false);
/// `--rerank`: re-score the candidate pool against the exact vectors before
/// taking the top K.
static RERANK: std::sync::atomic::AtomicBool = std::sync::atomic::AtomicBool::new(false);
fn storage() -> Storage {
if INT8.load(std::sync::atomic::Ordering::Relaxed) {
Storage::Int8
} else {
Storage::Float32
}
}
fn make_dataset(n: usize, seed: u64) -> Dataset {
let mut rng = Rng(seed);
if UNIFORM.load(std::sync::atomic::Ordering::Relaxed) {
@@ -169,6 +191,11 @@ fn text_for(cluster: usize, i: usize, rng: &mut Rng) -> String {
// Measurement helpers
// ---------------------------------------------------------------------------
/// Exact cosine distance between unit-length vectors.
fn exact_dist(a: &[f32], b: &[f32]) -> f32 {
1.0 - a.iter().zip(b).map(|(x, y)| x * y).sum::<f32>()
}
fn exact_top_k(vectors: &[Vec<f32>], query: &[f32], k: usize) -> Vec<usize> {
// Vectors are unit length, so cosine order == dot-product order.
let mut scored: Vec<(usize, f32)> = vectors
@@ -198,6 +225,83 @@ fn summarize(mut samples: Vec<Duration>) -> Latency {
}
}
/// Counts live heap bytes, so a structure's cost can be measured by
/// difference.
///
/// RSS cannot do this from inside one process: freeing a large structure
/// returns its pages to the allocator's pool rather than to the OS, so
/// allocating the next one shows no change. Measured that way, a store that
/// holds the corpus twice and one that holds it once look identical.
struct CountingAllocator;
static LIVE_BYTES: std::sync::atomic::AtomicI64 = std::sync::atomic::AtomicI64::new(0);
/// High-water mark of [`LIVE_BYTES`] since it was last reset.
///
/// Live bytes at a checkpoint cannot see a buffer that was allocated and
/// freed in between, and that is exactly the shape of a transient copy —
/// which still has to fit in memory while it exists.
static PEAK_BYTES: std::sync::atomic::AtomicI64 = std::sync::atomic::AtomicI64::new(0);
fn note_peak(live: i64) {
PEAK_BYTES.fetch_max(live, std::sync::atomic::Ordering::Relaxed);
}
// SAFETY: every method forwards to the system allocator with the same layout
// it was given, and only adds bookkeeping around it.
unsafe impl std::alloc::GlobalAlloc for CountingAllocator {
unsafe fn alloc(&self, layout: std::alloc::Layout) -> *mut u8 {
let ptr = unsafe { std::alloc::System.alloc(layout) };
if !ptr.is_null() {
let live = LIVE_BYTES
.fetch_add(layout.size() as i64, std::sync::atomic::Ordering::Relaxed)
+ layout.size() as i64;
note_peak(live);
}
ptr
}
unsafe fn dealloc(&self, ptr: *mut u8, layout: std::alloc::Layout) {
LIVE_BYTES.fetch_sub(layout.size() as i64, std::sync::atomic::Ordering::Relaxed);
unsafe { std::alloc::System.dealloc(ptr, layout) }
}
unsafe fn realloc(&self, ptr: *mut u8, layout: std::alloc::Layout, new_size: usize) -> *mut u8 {
let new_ptr = unsafe { std::alloc::System.realloc(ptr, layout, new_size) };
if !new_ptr.is_null() {
let delta = new_size as i64 - layout.size() as i64;
let live = LIVE_BYTES.fetch_add(delta, std::sync::atomic::Ordering::Relaxed) + delta;
note_peak(live);
}
new_ptr
}
}
#[global_allocator]
static ALLOCATOR: CountingAllocator = CountingAllocator;
/// Live heap bytes right now.
fn heap_bytes() -> u64 {
LIVE_BYTES.load(std::sync::atomic::Ordering::Relaxed).max(0) as u64
}
/// Start watching for a new high-water mark from the current live total.
fn reset_peak() {
PEAK_BYTES.store(
LIVE_BYTES.load(std::sync::atomic::Ordering::Relaxed),
std::sync::atomic::Ordering::Relaxed,
);
}
/// The highest live total seen since [`reset_peak`].
fn peak_bytes() -> u64 {
PEAK_BYTES.load(std::sync::atomic::Ordering::Relaxed).max(0) as u64
}
fn mib(bytes: u64) -> f64 {
bytes as f64 / (1 << 20) as f64
}
fn micros(d: Duration) -> f64 {
d.as_secs_f64() * 1e6
}
@@ -219,11 +323,12 @@ fn bench_ann(n: usize, json: &mut Vec<serde_json::Value>) {
.collect();
let started = Instant::now();
let index = HnswIndex::build_with_metric(
let index = HnswIndex::build_with(
&data.vectors,
HNSW_M,
HNSW_EF_CONSTRUCTION,
DistanceMetric::Cosine,
storage(),
);
let build = started.elapsed();
@@ -240,7 +345,8 @@ fn bench_ann(n: usize, json: &mut Vec<serde_json::Value>) {
);
println!(
"\n### HNSW, N = {n}, dim = {DIM}, M = {HNSW_M}, ef_construction = {HNSW_EF_CONSTRUCTION}\n"
"\n### HNSW, N = {n}, dim = {DIM}, M = {HNSW_M}, ef_construction = {HNSW_EF_CONSTRUCTION}, storage = {:?}\n",
index.storage()
);
println!(
"build: {:.1} ms ({:.0} vectors/s) · exact scan: {:.0} QPS, p50 {:.0} µs\n",
@@ -251,12 +357,26 @@ fn bench_ann(n: usize, json: &mut Vec<serde_json::Value>) {
);
println!("| ef | recall@{K} | QPS | p50 µs | p99 µs |");
println!("|---:|---:|---:|---:|---:|");
// With a quantised index the distances it returns are approximate, so
// the candidates are re-scored against the exact vectors the caller
// already holds (in the agent, the embedding cache) before taking the
// top K. `--rerank` prices that: it costs one exact distance per
// candidate and is what decides whether int8 is usable.
let rerank = RERANK.load(std::sync::atomic::Ordering::Relaxed);
let pool = if rerank { K * 4 } else { K };
for ef in EF_VALUES {
let mut hits = 0usize;
let mut samples = Vec::with_capacity(data.queries.len());
for (q, want) in data.queries.iter().zip(&truth) {
let t = Instant::now();
let got = index.search(q, K, ef);
let mut got = index.search(q, pool, ef.max(pool));
if rerank {
for cand in &mut got {
cand.1 = exact_dist(&data.vectors[cand.0], q);
}
got.select_nth_unstable_by(K - 1, |a, b| a.1.total_cmp(&b.1));
got.truncate(K);
}
samples.push(t.elapsed());
hits += got.iter().filter(|(id, _)| want.contains(id)).count();
}
@@ -369,6 +489,394 @@ fn bench_end_to_end(n: usize, json: &mut Vec<serde_json::Value>) {
}));
}
// ---------------------------------------------------------------------------
// Signing study: what does an Ed25519-signed checkpoint cost?
// ---------------------------------------------------------------------------
/// `--signing-study`: checkpoint time unsigned vs signed, `verify` time, and
/// the file-size cost of the stored per-record hashes. Default store
/// settings (float16, int8 index). Medians of five checkpoints / three
/// verifies.
fn signing_study(n: usize) {
use clawhdf5_agent::signing::SigningKey;
let data = make_dataset(n, 0x516 ^ n as u64);
let mut rng = Rng(9);
let entries: Vec<MemoryEntry> = data
.vectors
.iter()
.enumerate()
.map(|(i, v)| MemoryEntry {
chunk: text_for(data.cluster_of[i], i, &mut rng),
embedding: v.clone(),
source_channel: "bench".into(),
timestamp: i as f64,
session_id: format!("s{}", i % 50),
tags: format!("t{i}"),
})
.collect();
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("sign.h5");
let mut mem = HDF5Memory::create(MemoryConfig::new(path.clone(), "bench", DIM)).unwrap();
mem.save_batch(entries).unwrap();
std::hint::black_box(mem.hybrid_search(&data.queries[0], "", 1.0, 0.0, K));
let median = |mut v: Vec<Duration>| {
v.sort();
v[v.len() / 2]
};
let checkpoint = |mem: &mut HDF5Memory| {
median(
(0..5)
.map(|_| {
let t = Instant::now();
mem.flush_wal().unwrap();
t.elapsed()
})
.collect(),
)
};
let unsigned = checkpoint(&mut mem);
let unsigned_bytes = std::fs::metadata(&path).unwrap().len();
let key = SigningKey::from_bytes(&[7; 32]);
mem.set_signing_key(key.clone());
let signed = checkpoint(&mut mem);
let signed_bytes = std::fs::metadata(&path).unwrap().len();
drop(mem);
let vk = key.verifying_key();
let verify = median(
(0..3)
.map(|_| {
let t = Instant::now();
let r = HDF5Memory::verify(&path, &vk).unwrap();
let d = t.elapsed();
assert!(r.is_valid());
d
})
.collect(),
);
println!(
"| {n} | {:.1} | {:.1} | {:+.1} | {:.1} | {:+.2} |",
millis(unsigned),
millis(signed),
millis(signed) - millis(unsigned),
millis(verify),
(signed_bytes as f64 - unsigned_bytes as f64) / (1024.0 * 1024.0),
);
}
// ---------------------------------------------------------------------------
// Search options study: source filters, re-ranking, confidence rejection
// ---------------------------------------------------------------------------
/// `--options-study`: what `HDF5Memory::search`'s options cost and whether a
/// filtered search finds the right records. Filters keep 50%, 10% or 1% of
/// the store at random, or two whole clusters away from the query (the case
/// the index cannot serve, which falls back to an exact scan). Recall is
/// vector-only against an exact scan of the allowed records; latency is full
/// hybrid search. Hebbian boosting is off.
fn options_study(n: usize) {
use clawhdf5_agent::SearchOptions;
use clawhdf5_agent::confidence::ConfidenceConfig;
use clawhdf5_agent::hybrid::Fusion;
use clawhdf5_agent::reranker::ReRankConfig;
let data = make_dataset(n, 0x0B7 ^ n as u64);
let n_clusters = data.cluster_of.iter().max().map_or(1, |m| m + 1);
let mut rng = Rng(5);
let bucket_of: Vec<usize> = (0..n).map(|_| rng.below(100)).collect();
let bucket = &bucket_of;
let query_texts: Vec<String> = data
.query_cluster
.iter()
.enumerate()
.map(|(i, c)| text_for(*c, i, &mut rng))
.collect();
let exact_top = |q: &[f32], allowed: &dyn Fn(usize) -> bool| -> Vec<usize> {
let mut s: Vec<(usize, f32)> = (0..n)
.filter(|&i| allowed(i))
.map(|i| (i, data.vectors[i].iter().zip(q).map(|(a, b)| a * b).sum()))
.collect();
s.sort_by(|a, b| b.1.total_cmp(&a.1).then(a.0.cmp(&b.0)));
s.into_iter().take(K).map(|(i, _)| i).collect()
};
// Two stores: channel = random bucket, and channel = cluster.
let dir = tempfile::tempdir().unwrap();
let mut stores = Vec::new();
for by_cluster in [false, true] {
let mut rng = Rng(3);
let entries: Vec<MemoryEntry> = data
.vectors
.iter()
.enumerate()
.map(|(i, v)| MemoryEntry {
chunk: text_for(data.cluster_of[i], i, &mut rng),
embedding: v.clone(),
source_channel: if by_cluster {
format!("c{}", data.cluster_of[i])
} else {
format!("b{}", bucket[i])
},
timestamp: i as f64,
session_id: format!("s{}", i % 50),
tags: format!("t{i}"),
})
.collect();
let mut config = MemoryConfig::new(
dir.path().join(format!("opt_{by_cluster}.h5")),
"bench",
DIM,
);
config.hebbian_boost = 0.0;
let mut mem = HDF5Memory::create(config).unwrap();
mem.save_batch(entries).unwrap();
std::hint::black_box(mem.search(&data.queries[0], "", &SearchOptions::new(K)));
stores.push(mem);
}
let vector_only = SearchOptions::new(K).with_fusion(Fusion::Weighted {
vector: 1.0,
keyword: 0.0,
});
// (label, store, channels for query i, allowed(i, record))
type Case<'a> = (
String,
usize,
Box<dyn Fn(usize) -> Option<Vec<String>> + 'a>,
Box<dyn Fn(usize, usize) -> bool + 'a>,
);
let mut cases: Vec<Case> = vec![(
"no filter".into(),
0,
Box::new(|_| None),
Box::new(|_, _| true),
)];
for pct in [50usize, 10, 1] {
cases.push((
format!("random {pct}%"),
0,
Box::new(move |_| Some((0..pct).map(|b| format!("b{b}")).collect())),
Box::new(move |_, i| bucket[i] < pct),
));
}
let d = &data;
let away = move |qi: usize| {
let qc = d.query_cluster[qi];
[
(qc + n_clusters / 3) % n_clusters,
(qc + 2 * n_clusters / 3) % n_clusters,
]
};
cases.push((
"2 clusters away from the query".into(),
1,
Box::new(move |qi| Some(away(qi).iter().map(|c| format!("c{c}")).collect())),
Box::new(move |qi, i| away(qi).contains(&d.cluster_of[i])),
));
for (label, store, channels, allowed) in &cases {
let mem = &mut stores[*store];
let mut hits = 0;
let mut kept = 0;
for (qi, q) in data.queries.iter().enumerate() {
let mut opts = vector_only.clone();
opts.source_channels = channels(qi);
let got = mem.search(q, "", &opts);
let want = exact_top(q, &|i| allowed(qi, i));
kept += want.len();
hits += got.iter().filter(|r| want.contains(&r.index)).count();
}
let latency = summarize(
(0..N_QUERIES)
.map(|qi| {
let mut opts = SearchOptions::new(K);
opts.source_channels = channels(qi);
let t = Instant::now();
std::hint::black_box(mem.search(&data.queries[qi], &query_texts[qi], &opts));
t.elapsed()
})
.collect(),
);
println!(
"| {n} | {label} | {:.4} | {:.3} | {:.3} |",
hits as f64 / kept.max(1) as f64,
millis(latency.p50),
millis(latency.p99),
);
}
let mem = &mut stores[0];
for (label, opts) in [
(
"re-rank",
SearchOptions::new(K).with_rerank(ReRankConfig::default()),
),
(
"re-rank + confidence",
SearchOptions::new(K)
.with_rerank(ReRankConfig::default())
.with_confidence(ConfidenceConfig::default()),
),
] {
let latency = summarize(
(0..N_QUERIES)
.map(|qi| {
let t = Instant::now();
std::hint::black_box(mem.search(&data.queries[qi], &query_texts[qi], &opts));
t.elapsed()
})
.collect(),
);
println!(
"| {n} | {label} | — | {:.3} | {:.3} |",
millis(latency.p50),
millis(latency.p99)
);
}
}
// ---------------------------------------------------------------------------
// float16 study: what does half-precision embedding storage cost?
// ---------------------------------------------------------------------------
/// `--float16-study`: the same data in an `f32` store and a `float16` store.
/// Reports file size, checkpoint and open time, vector-search recall@10
/// against an exact scan of the *original* f32 vectors, how often the two
/// stores return the same top 10, and `hybrid_search` latency. Hebbian
/// boosting is off, so every query sees the same store.
fn float16_study(n: usize) {
let data = make_dataset(n, 0xF16 ^ n as u64);
let mut rng = Rng(11);
let query_texts: Vec<String> = data
.query_cluster
.iter()
.enumerate()
.map(|(i, c)| text_for(*c, i, &mut rng))
.collect();
// Exact top K by cosine (the vectors are unit length) on the f32 inputs.
let exact: Vec<Vec<usize>> = data
.queries
.iter()
.map(|q| {
let mut scored: Vec<(usize, f32)> = data
.vectors
.iter()
.enumerate()
.map(|(i, v)| (i, v.iter().zip(q).map(|(a, b)| a * b).sum()))
.collect();
scored.sort_by(|a, b| b.1.total_cmp(&a.1).then(a.0.cmp(&b.0)));
scored.into_iter().take(K).map(|(i, _)| i).collect()
})
.collect();
let dir = tempfile::tempdir().unwrap();
let mut per_variant: Vec<(bool, Vec<Vec<usize>>)> = Vec::new();
// `--f16-first` swaps the order, to check the numbers do not depend on
// which store runs first (page cache, allocator, CPU frequency).
let order = if F16_FIRST.load(std::sync::atomic::Ordering::Relaxed) {
[true, false]
} else {
[false, true]
};
for float16 in order {
let path = dir.path().join(format!("f16study_{float16}.h5"));
let mut rng = Rng(3);
let entries: Vec<MemoryEntry> = data
.vectors
.iter()
.enumerate()
.map(|(i, v)| MemoryEntry {
chunk: text_for(data.cluster_of[i], i, &mut rng),
embedding: v.clone(),
source_channel: "bench".into(),
timestamp: i as f64,
session_id: format!("s{}", i % 50),
tags: format!("t{i}"),
})
.collect();
let mut config = MemoryConfig::new(path.clone(), "bench", DIM);
config.float16 = float16;
config.hebbian_boost = 0.0;
let mut mem = HDF5Memory::create(config).unwrap();
mem.save_batch(entries).unwrap();
// Build the indexes, then time a checkpoint that writes everything.
std::hint::black_box(mem.hybrid_search(&data.queries[0], "", 1.0, 0.0, K));
let t = Instant::now();
mem.flush_wal().unwrap();
let checkpoint = t.elapsed();
drop(mem);
let file_bytes = std::fs::metadata(&path).unwrap().len();
// Median of three opens.
let mut opens: Vec<Duration> = (0..3)
.map(|_| {
let t = Instant::now();
let m = HDF5Memory::open(&path).unwrap();
let d = t.elapsed();
drop(m);
d
})
.collect();
opens.sort();
let mut mem = HDF5Memory::open(&path).unwrap();
// Vector-only search: empty text, all weight on the vector stage.
let results: Vec<Vec<usize>> = data
.queries
.iter()
.map(|q| {
mem.hybrid_search(q, "", 1.0, 0.0, K)
.iter()
.map(|r| r.index)
.collect()
})
.collect();
let hits: usize = results
.iter()
.zip(&exact)
.map(|(got, want)| got.iter().filter(|i| want.contains(i)).count())
.sum();
let recall = hits as f64 / (K * data.queries.len()) as f64;
let latency = summarize(
(0..N_QUERIES)
.map(|i| {
let t = Instant::now();
std::hint::black_box(mem.hybrid_search(
&data.queries[i],
&query_texts[i],
0.4,
0.6,
K,
));
t.elapsed()
})
.collect(),
);
let overlap = match per_variant.first() {
Some((_, other)) => {
let same: usize = results
.iter()
.zip(other)
.map(|(a, b)| a.iter().filter(|i| b.contains(i)).count())
.sum();
format!("{:.4}", same as f64 / (K * data.queries.len()) as f64)
}
None => "—".into(),
};
println!(
"| {n} | {} | {:.1} | {:.0} | {:.1} | {recall:.4} | {overlap} | {:.3} |",
if float16 { "float16" } else { "f32" },
mib(file_bytes),
millis(checkpoint),
millis(opens[1]),
millis(latency.p50),
);
per_variant.push((float16, results));
}
}
// ---------------------------------------------------------------------------
// Fusion study: does capping the keyword candidate pool change the ranking?
// ---------------------------------------------------------------------------
@@ -394,11 +902,12 @@ fn fusion_study(n: usize) {
.map(|(i, c)| text_for(*c, i, &mut rng))
.collect();
let bm25 = BM25Index::build(&texts, &vec![0u8; n]);
let index = HnswIndex::build_with_metric(
let index = HnswIndex::build_with(
&data.vectors,
HNSW_M,
HNSW_EF_CONSTRUCTION,
DistanceMetric::Cosine,
storage(),
);
let vec_pool = (K * 8).max(64);
@@ -450,6 +959,69 @@ fn fusion_study(n: usize) {
}
}
/// What an in-memory store costs, stage by stage. The vectors are the floor:
/// everything above it is bookkeeping that could in principle be shared.
fn bench_footprint(n: usize) {
let data = make_dataset(n, 0xF007 ^ n as u64);
let mut rng = Rng(11);
let dir = tempfile::TempDir::new().unwrap();
let path = dir.path().join("footprint.h5");
let base = heap_bytes();
let entries: Vec<MemoryEntry> = data
.vectors
.iter()
.enumerate()
.map(|(i, v)| MemoryEntry {
chunk: text_for(data.cluster_of[i], i, &mut rng),
embedding: v.clone(),
source_channel: "bench".into(),
timestamp: i as f64,
session_id: format!("s{}", i % 50),
tags: format!("t{i}"),
})
.collect();
let after_entries = heap_bytes();
let mut config = MemoryConfig::new(path, "bench", DIM);
config.quantized_index = INT8.load(std::sync::atomic::Ordering::Relaxed);
let mut mem = HDF5Memory::create(config).unwrap();
mem.save_batch(entries).unwrap();
let after_store = heap_bytes();
// First query builds the vector and keyword indexes.
std::hint::black_box(mem.hybrid_search(&data.queries[0], "record", 0.7, 0.3, K));
let after_indexes = heap_bytes();
// Reopening is the figure that matters for a long-lived process, and the
// only one RSS reports honestly: memory freed when the ingest buffers went
// away stays in the allocator's pool, so the stage deltas above understate
// what was given back.
let path = mem.config().path.clone();
drop(mem);
let before_open = heap_bytes();
reset_peak();
let reopened = HDF5Memory::open(&path).unwrap();
let after_open = heap_bytes();
let loaded = after_open.saturating_sub(before_open);
// Peak over the open, not just what it leaves behind: a buffer allocated
// and freed during the parse never shows up in the live total.
let peak = peak_bytes().saturating_sub(before_open);
drop(reopened);
let raw = (n * DIM * 4) as u64;
println!(
"| {n} | {:.0} | {:.0} | {:.0} | {:.0} | {:.0} | {:.0} | {:.2}x |",
mib(raw),
mib(after_entries.saturating_sub(base)),
mib(after_store.saturating_sub(after_entries)),
mib(after_indexes.saturating_sub(after_store)),
mib(loaded),
mib(peak),
loaded as f64 / raw as f64,
);
}
fn main() {
let args: Vec<String> = std::env::args().skip(1).collect();
let full = args.iter().any(|a| a == "--full");
@@ -464,6 +1036,60 @@ fn main() {
}
return;
}
if args.iter().any(|a| a == "--signing-study") {
println!("## Signed checkpoints ({DIM}-dim, float16, int8 index)\n");
println!(
"| N | checkpoint ms, unsigned | checkpoint ms, signed | signing adds ms | verify ms | file MiB added |"
);
println!("|---:|---:|---:|---:|---:|---:|");
for &n in if full {
&[1_000, 10_000, 100_000][..]
} else {
&[1_000, 10_000][..]
} {
signing_study(n);
}
return;
}
if args.iter().any(|a| a == "--options-study") {
println!("## Search options ({DIM}-dim, k = {K}, Hebbian boost off)\n");
println!("| N | options | filtered recall@10 | p50 ms | p99 ms |");
println!("|---:|---|---:|---:|---:|");
for &n in if full {
&[10_000, 100_000][..]
} else {
&[10_000][..]
} {
options_study(n);
}
return;
}
if args.iter().any(|a| a == "--f16-first") {
F16_FIRST.store(true, std::sync::atomic::Ordering::Relaxed);
}
if args.iter().any(|a| a == "--float16-study") {
println!("## float16 embedding storage ({DIM}-dim, int8 index, Hebbian boost off)\n");
println!(
"| N | embeddings | file MiB | checkpoint ms | open ms | recall@10 | top-10 overlap with the other | hybrid p50 ms |"
);
println!("|---:|---|---:|---:|---:|---:|---:|---:|");
for &n in if full {
&[1_000, 10_000, 100_000][..]
} else {
&[1_000, 10_000][..]
} {
float16_study(n);
}
return;
}
if args.iter().any(|a| a == "--int8") {
INT8.store(true, std::sync::atomic::Ordering::Relaxed);
println!("(int8-quantised index vectors)");
}
if args.iter().any(|a| a == "--rerank") {
RERANK.store(true, std::sync::atomic::Ordering::Relaxed);
println!("(candidates re-scored against exact vectors)");
}
if args.iter().any(|a| a == "--uniform") {
UNIFORM.store(true, std::sync::atomic::Ordering::Relaxed);
println!("(uniform random data)");
@@ -485,9 +1111,25 @@ fn main() {
let mut json = Vec::new();
println!("## Search harness");
if args.iter().any(|a| a == "--footprint") {
println!("\n### Resident memory, {DIM}-dim f32\n");
println!(
"| N | vectors (raw) | entries MiB | store MiB | indexes MiB | reopened MiB | peak during open MiB | reopened / raw |"
);
println!("|---:|---:|---:|---:|---:|---:|---:|---:|");
for &n in sizes {
bench_footprint(n);
}
return;
}
// `--e2e-only` skips the index benchmarks, so the end-to-end section runs
// in a process that has not already spun up a thread pool.
if !args.iter().any(|a| a == "--e2e-only") {
for &n in sizes {
bench_ann(n, &mut json);
}
}
if ann_only {
return;
crates/clawhdf5-bench/tests/concurrent_read_smoke.rs
+148
View File
@@ -0,0 +1,148 @@
//! Keeps the concurrent-read harnesses working: runs `concurrent_read`, the
//! h5py script (threads and processes) and the comparison script end to end
//! on tiny files. h5py reading the files also checks, element by element at
//! spot positions, that both harnesses generate the same data and slabs.
//!
//! The h5py half is skipped when python3 with h5py is unavailable, unless
//! `CLAWHDF5_REQUIRE_INTEROP=1`; `CLAWHDF5_PYTHON` picks the interpreter.
use std::path::{Path, PathBuf};
use std::process::Command;
fn python() -> String {
std::env::var("CLAWHDF5_PYTHON").unwrap_or_else(|_| "python3".to_string())
}
fn interop_required() -> bool {
std::env::var("CLAWHDF5_REQUIRE_INTEROP").is_ok_and(|v| v == "1")
}
fn python_available() -> bool {
Command::new(python())
.args(["-c", "import h5py, numpy"])
.output()
.map(|o| o.status.success())
.unwrap_or(false)
}
fn scripts() -> PathBuf {
Path::new(env!("CARGO_MANIFEST_DIR")).join("scripts")
}
fn run(cmd: &mut Command) -> String {
let out = cmd.output().expect("spawn");
assert!(
out.status.success(),
"{cmd:?} failed\nSTDOUT:\n{}\nSTDERR:\n{}",
String::from_utf8_lossy(&out.stdout),
String::from_utf8_lossy(&out.stderr)
);
String::from_utf8_lossy(&out.stdout).into_owned()
}
const SMALL: [&str; 8] = [
"--threads",
"1,2",
"--slabs",
"8",
"--reps",
"1",
"--slab",
"64",
];
fn results(path: &Path) -> serde_json::Value {
serde_json::from_str(&std::fs::read_to_string(path).unwrap()).unwrap()
}
#[test]
fn harnesses_run_end_to_end_on_tiny_files() {
let dir = tempfile::TempDir::new().unwrap();
let data = dir.path().join("data");
let claw = dir.path().join("claw.json");
let bin = env!("CARGO_BIN_EXE_concurrent_read");
run(Command::new(bin)
.arg("--dir")
.arg(&data)
.args(["--datasets", "3", "--mib", "1"])
.args(SMALL)
.arg("--json")
.arg(&claw));
// Second run reuses the files (and exercises --cold).
let out = Command::new(bin)
.arg("--dir")
.arg(&data)
.args(["--datasets", "3", "--mib", "1", "--cold"])
.args(SMALL)
.output()
.unwrap();
assert!(out.status.success());
assert!(String::from_utf8_lossy(&out.stderr).contains("reusing"));
let doc = results(&claw);
assert_eq!(doc["tool"], "clawhdf5");
// 2 layouts x 2 modes x 2 thread counts.
assert_eq!(doc["results"].as_array().unwrap().len(), 8);
for r in doc["results"].as_array().unwrap() {
assert!(r["mb_s"].as_f64().unwrap() > 0.0, "{r}");
}
if !python_available() {
assert!(
!interop_required(),
"CLAWHDF5_REQUIRE_INTEROP=1 but {} has no h5py",
python()
);
eprintln!("skipping the h5py half: no h5py in {}", python());
return;
}
let mut jsons = vec![claw];
for executor in ["threads", "processes"] {
let out = dir.path().join(format!("h5py-{executor}.json"));
run(Command::new(python())
.arg(scripts().join("concurrent_read_h5py.py"))
.arg("--dir")
.arg(&data)
.args(["--executor", executor])
.args(SMALL)
.arg("--json")
.arg(&out));
let doc = results(&out);
assert_eq!(doc["tool"], format!("h5py-{executor}"));
assert_eq!(doc["results"].as_array().unwrap().len(), 8);
jsons.push(out);
}
let table = run(Command::new(python())
.arg(scripts().join("compare_concurrent_read.py"))
.args(&jsons));
assert!(table.contains("| deflate | same | 2 |"), "{table}");
assert!(table.contains("clawhdf5 / h5py-processes"), "{table}");
// A different workload must not be compared.
let other = dir.path().join("other.json");
run(Command::new(python())
.arg(scripts().join("concurrent_read_h5py.py"))
.arg("--dir")
.arg(&data)
.args([
"--threads",
"1",
"--slabs",
"4",
"--reps",
"1",
"--slab",
"64",
])
.arg("--json")
.arg(&other));
let out = Command::new(python())
.arg(scripts().join("compare_concurrent_read.py"))
.arg(&jsons[0])
.arg(&other)
.output()
.unwrap();
assert!(!out.status.success());
assert!(String::from_utf8_lossy(&out.stderr).contains("slabs"));
}
crates/clawhdf5-cli/Cargo.toml
+3 -2
View File
@@ -1,7 +1,8 @@
[package]
name = "clawhdf5-cli"
version = "2.4.0"
version = "2.7.0"
edition = "2024"
rust-version.workspace = true
license = "MIT"
description = "CLI for clawhdf5 agent memory — create, save, search, recall, stats"
repository = "https://git.redclaw.dev/quantumclaw/clawhdf5"
@@ -14,7 +15,7 @@ name = "clawhdf5"
path = "src/main.rs"
[dependencies]
clawhdf5-agent = { path = "../clawhdf5-agent", version = "2.4.0" }
clawhdf5-agent = { path = "../clawhdf5-agent", version = "2.7.0" }
clap = { version = "4", features = ["derive", "env"] }
serde_json = "1"
serde = { workspace = true }
crates/clawhdf5-cli/src/main.rs
+165 -17
View File
@@ -1,15 +1,22 @@
use std::path::PathBuf;
use std::path::{Path, PathBuf};
use clap::{Parser, Subcommand};
use clawhdf5_agent::signing::{self, SigningKey, VerifyingKey};
use clawhdf5_agent::{AgentMemory, HDF5Memory, MemoryConfig, MemoryEntry};
/// ClawhDF5 — HDF5-backed cognitive memory for AI agents
#[derive(Parser)]
#[command(name = "clawhdf5", version, about)]
struct Cli {
/// Path to the .h5 memory file
/// Path to the .h5 memory file (not needed for `keygen`)
#[arg(short, long, env = "CLAWHDF5_PATH")]
path: PathBuf,
path: Option<PathBuf>,
/// File holding an Ed25519 signing key (64 hex characters, from
/// `keygen`). Every checkpoint this command makes is then signed; a
/// signed store refuses to checkpoint without it.
#[arg(long, env = "CLAWHDF5_SIGNING_KEY", global = true)]
signing_key: Option<PathBuf>,
#[command(subcommand)]
command: Commands,
@@ -28,6 +35,22 @@ enum Commands {
/// Enable write-ahead log
#[arg(long)]
wal: bool,
/// Hold the vector index's copy of the embeddings as f32 instead of
/// the default int8 (which uses a quarter of the memory and is faster
/// at equal recall)
#[arg(long)]
f32_index: bool,
/// Accepted for compatibility; int8 is now the default
#[arg(long, hide = true, conflicts_with = "f32_index")]
quantized_index: bool,
/// Store embeddings as full-precision f32 instead of the default
/// half precision (float16: half the bytes, about three significant
/// digits, values within ±65504)
#[arg(long)]
f32: bool,
/// Accepted for compatibility; float16 is now the default
#[arg(long, hide = true, conflicts_with = "f32")]
float16: bool,
},
/// Save a memory entry (reads JSON from stdin or --json)
Save {
@@ -75,6 +98,38 @@ enum Commands {
/// Destination path
dest: PathBuf,
},
/// Generate an Ed25519 signing key for signed checkpoints
Keygen {
/// Where to write the secret key (created new, owner-only on Unix)
#[arg(long)]
out: PathBuf,
},
/// Verify a signed store against a public key; exit status 2 if not valid
Verify {
/// The trusted public key: 64 hex characters, or a file holding them
#[arg(long)]
public_key: String,
},
}
fn read_signing_key(path: &Path) -> Result<SigningKey, Box<dyn std::error::Error>> {
let text = std::fs::read_to_string(path)
.map_err(|e| format!("cannot read signing key {}: {e}", path.display()))?;
let bytes = signing::from_hex::<32>(&text)
.ok_or_else(|| format!("{} is not a 64-hex-character key", path.display()))?;
Ok(SigningKey::from_bytes(&bytes))
}
/// Open for writing, with the signing key applied if one was given.
fn open_writable(
path: &Path,
key: &Option<SigningKey>,
) -> Result<HDF5Memory, Box<dyn std::error::Error>> {
let mut mem = HDF5Memory::open(path)?;
if let Some(k) = key {
mem.set_signing_key(k.clone());
}
Ok(mem)
}
fn main() {
@@ -87,17 +142,76 @@ fn main() {
}
fn run(cli: Cli) -> Result<(), Box<dyn std::error::Error>> {
if let Commands::Keygen { out } = &cli.command {
let key = signing::generate_key();
let mut opts = std::fs::OpenOptions::new();
opts.write(true).create_new(true);
#[cfg(unix)]
{
use std::os::unix::fs::OpenOptionsExt;
opts.mode(0o600);
}
use std::io::Write;
let mut f = opts
.open(out)
.map_err(|e| format!("cannot create {}: {e}", out.display()))?;
writeln!(f, "{}", signing::to_hex(&key.to_bytes()))?;
let j = serde_json::json!({
"status": "generated",
"secret_key_file": out.display().to_string(),
"public_key": signing::to_hex(&key.verifying_key().to_bytes()),
});
println!("{}", serde_json::to_string_pretty(&j)?);
return Ok(());
}
let path = cli
.path
.clone()
.ok_or("--path (or CLAWHDF5_PATH) is required")?;
let key = cli
.signing_key
.as_deref()
.map(read_signing_key)
.transpose()?;
match cli.command {
Commands::Create { agent_id, dim, wal } => {
let mut config = MemoryConfig::new(cli.path.clone(), &agent_id, dim);
Commands::Create {
agent_id,
dim,
wal,
f32_index,
quantized_index: _,
f32,
float16: _,
} => {
let mut config = MemoryConfig::new(path.clone(), &agent_id, dim);
config.wal_enabled = wal;
let mem = HDF5Memory::create(config)?;
// As with --f32-index: only ever switch the library default off.
if f32 {
config.float16 = false;
}
let config_float16 = config.float16;
// Only ever switch *off* the library default: assigning the flag
// outright would force every CLI-created store back to f32 unless
// the caller knew to ask for int8.
if f32_index {
config.quantized_index = false;
}
let config_quantized = config.quantized_index;
let mut mem = HDF5Memory::create(config)?;
// Sign straight away, so the store is never on disk unsigned.
if let Some(k) = &key {
mem.set_signing_key(k.clone());
mem.flush_wal()?;
}
let j = serde_json::json!({
"status": "created",
"path": cli.path.display().to_string(),
"path": path.display().to_string(),
"agent_id": agent_id,
"embedding_dim": dim,
"wal_enabled": wal,
"quantized_index": config_quantized,
"float16": config_float16,
"signed": mem.is_signed(),
"count": mem.count(),
});
println!("{}", serde_json::to_string_pretty(&j)?);
@@ -114,7 +228,7 @@ fn run(cli: Cli) -> Result<(), Box<dyn std::error::Error>> {
}
};
let entry: MemoryEntry = serde_json::from_str(&input)?;
let mut mem = HDF5Memory::open(&cli.path)?;
let mut mem = open_writable(&path, &key)?;
let idx = mem.save(entry)?;
let j = serde_json::json!({ "status": "saved", "index": idx, "count": mem.count() });
println!("{}", serde_json::to_string(&j)?);
@@ -128,7 +242,7 @@ fn run(cli: Cli) -> Result<(), Box<dyn std::error::Error>> {
keyword_weight,
} => {
let emb: Vec<f32> = serde_json::from_str(&embedding)?;
let mut mem = HDF5Memory::open(&cli.path)?;
let mut mem = open_writable(&path, &key)?;
let results = mem.hybrid_search(&emb, &query, vector_weight, keyword_weight, top_k);
let j: Vec<serde_json::Value> = results
.iter()
@@ -146,7 +260,7 @@ fn run(cli: Cli) -> Result<(), Box<dyn std::error::Error>> {
}
Commands::Recall { index } => {
let mem = HDF5Memory::open_read_only(&cli.path)?;
let mem = HDF5Memory::open_read_only(&path)?;
match mem.get_chunk(index) {
Some(content) => {
let j = serde_json::json!({ "index": index, "chunk": content });
@@ -160,22 +274,23 @@ fn run(cli: Cli) -> Result<(), Box<dyn std::error::Error>> {
}
Commands::Stats => {
let mem = HDF5Memory::open_read_only(&cli.path)?;
let mem = HDF5Memory::open_read_only(&path)?;
let cfg = mem.config();
let j = serde_json::json!({
"path": cli.path.display().to_string(),
"path": path.display().to_string(),
"agent_id": cfg.agent_id,
"embedding_dim": cfg.embedding_dim,
"count": mem.count(),
"active": mem.count_active(),
"wal_enabled": cfg.wal_enabled,
"wal_pending": mem.wal_pending_count(),
"signed": mem.is_signed(),
});
println!("{}", serde_json::to_string_pretty(&j)?);
}
Commands::FlushWal => {
let mut mem = HDF5Memory::open(&cli.path)?;
let mut mem = open_writable(&path, &key)?;
let before = mem.wal_pending_count();
mem.flush_wal()?;
let j = serde_json::json!({
@@ -187,7 +302,7 @@ fn run(cli: Cli) -> Result<(), Box<dyn std::error::Error>> {
}
Commands::AgentsMd { output } => {
let mem = HDF5Memory::open_read_only(&cli.path)?;
let mem = HDF5Memory::open_read_only(&path)?;
let md = mem.generate_agents_md();
match output {
Some(p) => {
@@ -199,7 +314,7 @@ fn run(cli: Cli) -> Result<(), Box<dyn std::error::Error>> {
}
Commands::Export => {
let mem = HDF5Memory::open_read_only(&cli.path)?;
let mem = HDF5Memory::open_read_only(&path)?;
for i in 0..mem.count() {
if let Some(chunk) = mem.get_chunk(i) {
let j = serde_json::json!({ "index": i, "chunk": chunk });
@@ -208,11 +323,44 @@ fn run(cli: Cli) -> Result<(), Box<dyn std::error::Error>> {
}
}
Commands::Keygen { .. } => unreachable!("handled before opening a store"),
Commands::Verify { public_key } => {
let text = if Path::new(&public_key).is_file() {
std::fs::read_to_string(&public_key)?
} else {
public_key
};
let bytes = signing::from_hex::<32>(&text)
.ok_or("--public-key must be 64 hex characters or a file holding them")?;
let trusted = VerifyingKey::from_bytes(&bytes)?;
let r = HDF5Memory::verify(&path, &trusted)?;
let j = serde_json::json!({
"valid": r.is_valid(),
"signed": r.signed,
"key_matches": r.key_matches,
"signature_valid": r.signature_valid,
"records_match": r.records_match,
"settings_match": r.settings_match,
"sessions_match": r.sessions_match,
"graph_match": r.graph_match,
"changed_records": r.changed_records,
"record_count": r.record_count,
"signed_record_count": r.signed_record_count,
"signed_by": r.public_key.map(|k| signing::to_hex(&k)),
"wal_entries_unsigned": r.wal_entries_unsigned,
});
println!("{}", serde_json::to_string_pretty(&j)?);
if !r.is_valid() {
std::process::exit(2);
}
}
Commands::Snapshot { dest } => {
let _result = clawhdf5_agent::storage::snapshot_file(&cli.path, &dest)?;
let _result = clawhdf5_agent::storage::snapshot_file(&path, &dest)?;
let j = serde_json::json!({
"status": "snapshot_created",
"source": cli.path.display().to_string(),
"source": path.display().to_string(),
"dest": dest.display().to_string(),
});
println!("{}", serde_json::to_string(&j)?);
crates/clawhdf5-derive/Cargo.toml
+2 -1
View File
@@ -1,7 +1,8 @@
[package]
name = "clawhdf5-derive"
version = "2.4.0"
version = "2.7.0"
edition = "2024"
rust-version.workspace = true
description = "Derive macros for rustyhdf5 HDF5 traits"
license = "MIT"
repository = "https://git.redclaw.dev/quantumclaw/clawhdf5"
crates/clawhdf5-filters/Cargo.toml
+8 -3
View File
@@ -1,7 +1,8 @@
[package]
name = "clawhdf5-filters"
version = "2.4.0"
version = "2.7.0"
edition = "2024"
rust-version.workspace = true
description = "Filter and compression pipeline for clawhdf5"
license = "MIT"
repository = "https://git.redclaw.dev/quantumclaw/clawhdf5"
@@ -25,8 +26,12 @@ name = "compression_bench"
harness = false
[features]
default = ["fast-deflate"]
# Pure-Rust zlib-rs by default; `fast-deflate` (zlib-ng, C) overrides it.
default = ["zlib-rs"]
fast-deflate = ["flate2/zlib-ng"]
system-zlib = ["flate2/zlib-default"]
zlib-rs = ["flate2/zlib-rs"]
# `runtime_detection` gives zlib-rs `std`, which it needs to detect and use
# SIMD at runtime. flate2 enables it by default, but we build flate2 with
# default-features = false, and without it zlib-rs inflates 3.5x slower.
zlib-rs = ["flate2/zlib-rs", "flate2/runtime_detection"]
apple-compression = []
crates/clawhdf5-filters/README.md
+6 -4
View File
@@ -8,16 +8,18 @@ Filter and compression pipeline for clawhdf5.
## Features
- DEFLATE compression/decompression
- Fast deflate via zlib-ng (`fast-deflate` feature)
- Pure-Rust deflate via zlib-rs (default, `zlib-rs` feature)
- zlib-ng instead, if you want it (`fast-deflate` feature; C, needs cmake)
- Apple Compression framework support (`apple-compression` feature)
## Usage
```rust
use clawhdf5_filters::{deflate_decode, deflate_encode};
use clawhdf5_filters::{deflate_compress, deflate_decompress};
let compressed = deflate_encode(&data, 6).unwrap();
let decompressed = deflate_decode(&compressed).unwrap();
let compressed = deflate_compress(&data, 6).unwrap();
// The second argument bounds the output: the expected decompressed size.
let decompressed = deflate_decompress(&compressed, data.len()).unwrap();
```
## License
crates/clawhdf5-filters/src/fast_deflate.rs
+111 -48
View File
@@ -1,12 +1,13 @@
//! Fast deflate backends: Apple Compression Framework and zlib-ng.
//! Deflate backends: Apple Compression Framework, zlib-ng and zlib-rs.
//!
//! Backend selection priority (decompression & compression):
//! 1. Apple Compression Framework (macOS only, `apple-compression` feature)
//! 2. flate2 with zlib-ng backend (`fast-deflate` feature) or miniz_oxide (default)
//! 2. flate2 with zlib-ng (`fast-deflate`), else zlib-rs (`zlib-rs`, the
//! default), else miniz_oxide
//!
//! The Apple Compression Framework uses hardware-accelerated zlib on Apple Silicon
//! and is typically the fastest option on macOS. zlib-ng is the fastest portable
//! option and what C HDF5 uses internally.
//! and is typically the fastest option on macOS. zlib-rs is a pure-Rust port of
//! zlib-ng; see `BENCHMARKS.md` for how the two compare.
// ---------------------------------------------------------------------------
// Apple Compression Framework FFI (macOS only)
@@ -243,65 +244,117 @@ mod apple {
}
// ---------------------------------------------------------------------------
// Streaming decompression via flate2 (uses zlib-ng when fast-deflate enabled)
// One-shot (de)compression via flate2 (whichever backend flate2 was built with)
//
// The whole input goes to the codec in one call, into an output buffer sized
// up front. `flate2::read::ZlibDecoder` / `write::ZlibEncoder` stream through a
// 32 KiB buffer instead, which cost zlib-rs up to 3.7x against zlib-ng on a
// 1 MB chunk. clawhdf5-format's deflate filter does the same; see
// `BENCHMARKS.md`, "Deflate backend".
// ---------------------------------------------------------------------------
/// Streaming decompress with pre-allocated output buffer.
///
/// When the output size is known (typical for HDF5 chunks), this avoids
/// dynamic reallocation by writing directly into a pre-sized buffer.
/// Decompress into a buffer pre-sized to `output_size`, the expected
/// decompressed length (known for HDF5 chunks). Output longer than that is an
/// error, as is a stream that ends early.
pub(crate) fn flate2_decompress_preallocated(
data: &[u8],
output_size: usize,
) -> Result<Vec<u8>, String> {
use std::io::Read;
let mut decoder = flate2::read::ZlibDecoder::new(data);
let mut output = vec![0u8; output_size];
let mut total_read = 0;
loop {
match decoder.read(&mut output[total_read..]) {
Ok(0) => break,
Ok(n) => total_read += n,
Err(e) => return Err(e.to_string()),
}
}
output.truncate(total_read);
Ok(output)
inflate_bounded(data, output_size, output_size)
}
/// Absolute ceiling on decompressed output when the caller has no size hint,
/// preventing unbounded allocation from a hostile/corrupted zlib stream.
const MAX_DECOMPRESS_SIZE: usize = 256 * 1024 * 1024;
/// Streaming decompress with dynamic sizing (when output size is unknown).
///
/// Bounded by [`MAX_DECOMPRESS_SIZE`] since there is no chunk-size hint to
/// validate against here — an unbounded `read_to_end` would let a hostile
/// zlib stream force arbitrarily large allocation (a "zlib bomb").
/// Decompress with no size hint, bounded by [`MAX_DECOMPRESS_SIZE`] so a
/// hostile zlib stream cannot force arbitrarily large allocation (a "zlib
/// bomb").
pub(crate) fn flate2_decompress_streaming(data: &[u8]) -> Result<Vec<u8>, String> {
use std::io::Read;
let decoder = flate2::read::ZlibDecoder::new(data);
let mut result = Vec::new();
decoder
.take(MAX_DECOMPRESS_SIZE as u64 + 1)
.read_to_end(&mut result)
.map_err(|e| e.to_string())?;
if result.len() > MAX_DECOMPRESS_SIZE {
return Err(format!(
let hint = data.len().saturating_mul(4).min(1 << 20);
inflate_bounded(data, hint, MAX_DECOMPRESS_SIZE).map_err(|e| {
if e.ends_with("exceeds size limit") {
format!(
"decompressed output exceeds {} MiB limit",
MAX_DECOMPRESS_SIZE / 1024 / 1024
));
)
} else {
e
}
Ok(result)
})
}
/// Compress data using flate2 (zlib-ng when fast-deflate enabled, else miniz_oxide).
/// Inflate a zlib stream, starting from `size_hint` bytes of output and
/// failing past `limit`.
fn inflate_bounded(data: &[u8], size_hint: usize, limit: usize) -> Result<Vec<u8>, String> {
use flate2::{Decompress, FlushDecompress, Status};
// One byte of headroom past the limit distinguishes an over-size stream
// from one that legitimately ends exactly at the limit.
let max_capacity = limit.saturating_add(1);
let mut out = Vec::new();
out.try_reserve_exact(size_hint.clamp(1, max_capacity))
.map_err(|e| format!("deflate: cannot allocate output: {e}"))?;
let mut inflater = Decompress::new(true);
loop {
let (in_before, out_before) = (inflater.total_in(), inflater.total_out());
let status = inflater
.decompress_vec(
&data[in_before as usize..],
&mut out,
FlushDecompress::Finish,
)
.map_err(|e| format!("deflate: {e}"))?;
if out.len() > limit {
return Err("deflate: output exceeds size limit".into());
}
match status {
Status::StreamEnd => return Ok(out),
Status::Ok | Status::BufError if out.len() == out.capacity() => {
let grow = out.capacity().min(max_capacity - out.capacity()).max(1);
out.try_reserve_exact(grow)
.map_err(|e| format!("deflate: cannot allocate output: {e}"))?;
}
Status::Ok | Status::BufError => {
if inflater.total_in() as usize >= data.len()
|| (inflater.total_in(), inflater.total_out()) == (in_before, out_before)
{
return Err("deflate: truncated stream".into());
}
}
}
}
}
/// Compress data using flate2 (zlib-ng, zlib-rs or miniz_oxide; see module docs).
pub(crate) fn flate2_compress(data: &[u8], level: u32) -> Result<Vec<u8>, String> {
use std::io::Write;
let mut encoder = flate2::write::ZlibEncoder::new(Vec::new(), flate2::Compression::new(level));
encoder.write_all(data).map_err(|e| e.to_string())?;
encoder.finish().map_err(|e| e.to_string())
use flate2::{Compress, Compression, FlushCompress, Status};
// zlib's compressBound, plus the zlib header and trailer.
let bound = data.len() + (data.len() >> 12) + (data.len() >> 14) + (data.len() >> 25) + 13 + 6;
let mut out = Vec::new();
out.try_reserve_exact(bound)
.map_err(|e| format!("deflate: cannot allocate output: {e}"))?;
let mut deflater = Compress::new(Compression::new(level), true);
loop {
let (in_before, out_before) = (deflater.total_in(), deflater.total_out());
let status = deflater
.compress_vec(&data[in_before as usize..], &mut out, FlushCompress::Finish)
.map_err(|e| format!("deflate: {e}"))?;
match status {
Status::StreamEnd => return Ok(out),
Status::Ok | Status::BufError if out.len() == out.capacity() => out
.try_reserve(out.capacity().max(4096))
.map_err(|e| format!("deflate: cannot allocate output: {e}"))?,
Status::Ok | Status::BufError => {
if (deflater.total_in(), deflater.total_out()) == (in_before, out_before) {
return Err("deflate: encoder made no progress".into());
}
}
}
}
}
// ---------------------------------------------------------------------------
@@ -312,7 +365,7 @@ pub(crate) fn flate2_compress(data: &[u8], level: u32) -> Result<Vec<u8>, String
///
/// Selection order:
/// 1. Apple Compression Framework (macOS + `apple-compression` feature)
/// 2. flate2 (zlib-ng with `fast-deflate`, otherwise miniz_oxide)
/// 2. flate2 (zlib-ng with `fast-deflate`, else zlib-rs, else miniz_oxide)
///
/// When `output_hint` > 0, pre-allocates the output buffer for zero-copy
/// decompression (avoids reallocation).
@@ -344,7 +397,7 @@ pub fn decompress(data: &[u8], output_hint: usize) -> Result<Vec<u8>, String> {
///
/// Selection order:
/// 1. Apple Compression Framework (macOS + `apple-compression` feature)
/// 2. flate2 (zlib-ng with `fast-deflate`, otherwise miniz_oxide)
/// 2. flate2 (zlib-ng with `fast-deflate`, else zlib-rs, else miniz_oxide)
pub fn compress(data: &[u8], level: u32) -> Result<Vec<u8>, String> {
#[cfg(all(target_os = "macos", feature = "apple-compression"))]
{
@@ -377,9 +430,19 @@ pub fn active_backend() -> &'static str {
{
"zlib-ng"
}
// flate2 prefers a C zlib over zlib-rs when both are enabled.
#[cfg(all(
not(all(target_os = "macos", feature = "apple-compression")),
not(feature = "fast-deflate"),
feature = "zlib-rs"
))]
{
"zlib-rs"
}
#[cfg(not(any(
all(target_os = "macos", feature = "apple-compression"),
feature = "fast-deflate"
feature = "fast-deflate",
feature = "zlib-rs"
)))]
{
"miniz_oxide"
@@ -436,7 +499,7 @@ mod tests {
fn backend_name_is_set() {
let name = active_backend();
assert!(
["miniz_oxide", "zlib-ng", "apple-compression"].contains(&name),
["miniz_oxide", "zlib-rs", "zlib-ng", "apple-compression"].contains(&name),
"unexpected backend: {name}"
);
}
crates/clawhdf5-filters/src/lib.rs
+6 -4
View File
@@ -2,12 +2,14 @@
//!
//! Provides deflate (zlib) decompression/compression with multiple backend options:
//!
//! - **Default**: `miniz_oxide` (pure Rust, no C dependencies)
//! - **`fast-deflate` feature**: `zlib-ng` via flate2 (~2-3x faster, matches C HDF5)
//! - **Default (`zlib-rs` feature)**: `zlib-rs` via flate2 (pure Rust, no C
//! dependencies)
//! - **`fast-deflate` feature**: `zlib-ng` via flate2 (C, built with cmake)
//! - **`apple-compression` feature**: Apple Compression Framework on macOS
//! (hardware-accelerated on Apple Silicon)
//! - With none of the above: `miniz_oxide` (pure Rust, slower)
//!
//! Backend priority: apple-compression > zlib-ng > miniz_oxide.
//! Backend priority: apple-compression > zlib-ng > zlib-rs > miniz_oxide.
pub mod fast_deflate;
@@ -115,7 +117,7 @@ mod tests {
fn backend_reports_name() {
let name = deflate_backend();
assert!(
["miniz_oxide", "zlib-ng", "apple-compression"].contains(&name),
["miniz_oxide", "zlib-rs", "zlib-ng", "apple-compression"].contains(&name),
"unexpected backend: {name}"
);
}
crates/clawhdf5-format/Cargo.toml
+42 -4
View File
@@ -1,7 +1,8 @@
[package]
name = "clawhdf5-format"
version = "2.4.0"
version = "2.7.0"
edition = "2024"
rust-version.workspace = true
description = "Pure-Rust HDF5 binary format parsing and writing — no C dependencies"
license = "MIT"
repository = "https://git.redclaw.dev/quantumclaw/clawhdf5"
@@ -21,18 +22,33 @@ zstd = { version = "0.13", optional = true }
blake3 = { version = "1", optional = true }
libaec-sys = { path = "../libaec-sys", version = "0.1", optional = true }
pco = { version = "1.0", optional = true }
# Pure-Rust Zstandard, for the plugin filters that embed zstd (bitshuffle,
# blosc). The `zstd` feature (filter 32015) links libzstd instead.
ruzstd = { version = "0.9", optional = true }
# bzip2 with its default backend, libbz2-rs-sys: a pure-Rust port of
# libbzip2 (no C is compiled, despite the -sys name).
bzip2 = { version = "0.6", optional = true }
snap = { version = "1", optional = true }
[target.'cfg(target_os = "linux")'.dependencies]
# madvise(MADV_HUGEPAGE) for large read buffers (see src/bulk_alloc.rs).
libc = { version = "0.2", default-features = false }
[dev-dependencies]
half = { workspace = true }
serde_json = "1"
criterion = { workspace = true }
clawhdf5-derive = { path = "../clawhdf5-derive", version = "2.4.0" }
clawhdf5-derive = { path = "../clawhdf5-derive", version = "2.7.0" }
[[bench]]
name = "bench"
harness = false
[features]
default = ["std", "checksum", "deflate", "provenance", "fast-deflate", "system-zlib-decompress"]
# Deflate backend: `zlib-rs` (pure Rust) by default. `fast-deflate` selects
# zlib-ng instead (C, built with cmake); flate2 prefers a C zlib whenever one
# is enabled, so turning it on anywhere in the build overrides the default.
default = ["std", "checksum", "deflate", "provenance", "zlib-rs", "system-zlib-decompress", "lzf"]
std = []
checksum = []
deflate = ["flate2"]
@@ -42,12 +58,34 @@ fast-checksum = ["crc32fast"]
fast-deflate = ["flate2/zlib-ng"]
system-zlib = ["flate2/zlib-default"]
system-zlib-decompress = []
zlib-rs = ["flate2/zlib-rs"]
# `runtime_detection` gives zlib-rs `std`, which it needs to detect and use
# SIMD at runtime. flate2 enables it by default, but we build flate2 with
# default-features = false, and without it zlib-rs inflates 3.5x slower.
zlib-rs = ["flate2/zlib-rs", "flate2/runtime_detection"]
lz4 = ["lz4_flex"]
zstd = ["dep:zstd"]
blake3_hash = ["blake3"]
szip = ["libaec-sys"]
pcodec = ["dep:pco"]
# Plugin filters, pure Rust. LZF (32000) is h5py's built-in compression; it
# has no dependencies, so it is on by default.
lzf = []
# Bitshuffle (32008), with its LZ4 and Zstandard modes.
bitshuffle = ["lz4_flex", "ruzstd"]
# bzip2 (307).
bzip2 = ["dep:bzip2", "std"]
# Blosc 1 (32001) with its BloscLZ, LZ4, Snappy, Zlib and Zstandard codecs.
blosc = ["lz4_flex", "ruzstd", "snap", "deflate", "std"]
# Blosc2 (32026), read-only: frames, B2ND arrays, and the Blosc codecs above.
blosc2 = ["blosc"]
# ZFP (32013, H5Z-ZFP), read-only: every mode, for int32, int64, float and
# double fields of 1 to 4 dimensions.
zfp = []
# Every plugin filter above.
plugin-filters = ["lzf", "bitshuffle", "bzip2", "blosc", "blosc2", "zfp"]
# Test instrumentation: per-thread counts of heap objects read (see
# `lookup_stats`), so tests can bound the cost of a name lookup.
lookup-stats = ["std"]
[[bench]]
name = "parallel_decompress_bench"
crates/clawhdf5-format/fuzz/.gitignore
+3
View File
@@ -1 +1,4 @@
target/
corpus/
artifacts/
coverage/
crates/clawhdf5-format/fuzz/corpus/fuzz_btree_v2/regression-crash-f98c19dc
BIN
View File
Binary file not shown.
crates/clawhdf5-format/fuzz/fuzz_targets/fuzz_btree_v2.rs
+27 -6
View File
@@ -1,15 +1,36 @@
#![no_main]
use clawhdf5_format::btree_v2::{BTreeV2Header, collect_btree_v2_records};
use libfuzzer_sys::fuzz_target;
fuzz_target!(|data: &[u8]| {
for &offset_size in &[4u8, 8] {
for &length_size in &[4u8, 8] {
let _ = clawhdf5_format::btree_v2::BTreeV2Header::parse(
data,
0,
offset_size,
length_size,
);
if let Ok(header) = BTreeV2Header::parse(data, 0, offset_size, length_size) {
let _ = collect_btree_v2_records(data, &header, offset_size, length_size);
}
}
}
// Parsing a header requires a valid checksum, which random input almost
// never has, so the traversal behind it went unfuzzed — and that is where
// a node listing itself as its own child overflowed the stack. Take the
// header fields straight from the input instead and walk the rest.
let Some((fields, file)) = data.split_first_chunk::<20>() else {
return;
};
let header = BTreeV2Header {
tree_type: fields[0],
node_size: u32::from_le_bytes([fields[1], fields[2], fields[3], fields[4]]),
record_size: u16::from_le_bytes([fields[5], fields[6]]),
depth: u16::from_le_bytes([fields[7], fields[8]]),
root_node_address: u64::from(u32::from_le_bytes([
fields[9], fields[10], fields[11], fields[12],
])),
num_records_in_root: u16::from_le_bytes([fields[13], fields[14]]),
total_records: u64::from(u32::from_le_bytes([
fields[15], fields[16], fields[17], fields[18],
])),
};
let offset_size = if fields[19] & 1 == 0 { 4 } else { 8 };
let _ = collect_btree_v2_records(file, &header, offset_size, 8);
});
crates/clawhdf5-format/src/addr.rs
+121
View File
@@ -0,0 +1,121 @@
//! File address and length → in-memory index conversion.
//!
//! HDF5 addresses and lengths are 64-bit; the file is parsed through a
//! `&[u8]` indexed by `usize`. On a 64-bit target every `u64` fits, but on a
//! 32-bit one (`wasm32`, `i686`, `thumbv7em`) an address past `usize::MAX`
//! used to be truncated by an `as usize` cast — silently pointing at another
//! part of the file — or to panic. [`to_usize`] is the one conversion the
//! parsers use instead: such an address is a clean
//! [`FormatError::Overflow`]. It cannot be inside the data anyway: no slice
//! is longer than `isize::MAX` bytes.
#[cfg(not(feature = "std"))]
use alloc::format;
use crate::error::FormatError;
/// A file address, offset or length from the file as a `usize` index.
///
/// Fails with [`FormatError::Overflow`] when the value does not fit this
/// platform's `usize` (only possible on targets narrower than 64 bits).
#[inline]
pub fn to_usize(value: u64) -> Result<usize, FormatError> {
to_index::<usize>(value)
}
/// A file address for a [`crate::storage::Storage`] read, checked as
/// [`to_usize`] checks it: the parsers read through 64-bit offsets, but an
/// address that could not index an in-memory file on this platform is the
/// same [`FormatError::Overflow`] the slice parsers gave for it.
#[inline]
pub fn checked_addr(value: u64) -> Result<u64, FormatError> {
to_usize(value).map(|_| value)
}
/// [`to_usize`] for an index type of any width. `usize` is 64 bits wide on
/// the hosts CI tests on, where the error path cannot be reached through
/// `usize`; tests run the same code with `u32` in its place, as on a 32-bit
/// target.
#[inline]
fn to_index<T: TryFrom<u64>>(value: u64) -> Result<T, FormatError> {
T::try_from(value).map_err(|_| too_large(value))
}
/// A count or offset into an in-memory buffer (a codec's progress counter,
/// a size the writer computed from data it holds) as a `usize`, saturating
/// at `usize::MAX` instead of truncating.
///
/// For values that are bounded by the length of something in memory, so
/// always fit; if one ever did not, a saturated index fails its bounds check
/// or allocation instead of silently addressing the wrong bytes. A value
/// read from the file uses [`to_usize`].
#[inline]
pub fn saturating_usize(value: u64) -> usize {
saturating_index(value, usize::MAX)
}
/// [`saturating_usize`] for an index type of any width, whose largest
/// value is `max` (see [`to_index`]).
#[inline]
fn saturating_index<T: TryFrom<u64>>(value: u64, max: T) -> T {
T::try_from(value).unwrap_or(max)
}
#[cold]
#[inline(never)]
fn too_large(value: u64) -> FormatError {
FormatError::Overflow(format!(
"file address or length {value:#x} exceeds this platform's address space"
))
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn values_that_fit_convert_exactly() {
assert_eq!(to_usize(0), Ok(0));
assert_eq!(to_usize(0x1234), Ok(0x1234));
assert_eq!(to_usize(usize::MAX as u64), Ok(usize::MAX));
}
#[test]
fn saturating_conversion_never_wraps() {
assert_eq!(saturating_usize(0), 0);
assert_eq!(saturating_usize(0x1234), 0x1234);
assert_eq!(saturating_usize(usize::MAX as u64), usize::MAX);
// Past usize::MAX (32-bit targets) or at u64::MAX: saturates.
assert_eq!(saturating_usize(u64::MAX), usize::MAX);
}
#[test]
fn values_past_usize_max_are_an_error_not_truncated() {
// Reachable through `usize` only where it is narrower than u64 (no
// such target runs tests in CI), so the same conversion is run with
// u32 standing in for a 32-bit usize.
let max = u64::from(u32::MAX);
assert_eq!(to_index::<u32>(max), Ok(u32::MAX));
for past in [max + 1, max + 0x10, 0x1_0000_1234, u64::MAX] {
let err = to_index::<u32>(past).unwrap_err();
assert!(
matches!(err, FormatError::Overflow(_)),
"{past:#x}: {err:?}"
);
}
// Where an `as` cast would have wrapped to a small, valid-looking
// index, it is not returned.
assert_eq!(0x1_0000_1234_u64 as u32, 0x1234);
assert!(to_index::<u32>(0x1_0000_1234).is_err());
assert_eq!(saturating_index(max + 1, u32::MAX), u32::MAX);
assert_eq!(saturating_index(0x1_0000_1234, u32::MAX), u32::MAX);
assert_eq!(saturating_index(0x1234, u32::MAX), 0x1234);
// And through `usize` itself, whichever width it has here.
match (usize::MAX as u64).checked_add(1) {
Some(past) => assert!(matches!(to_usize(past), Err(FormatError::Overflow(_)))),
None => assert_eq!(to_usize(u64::MAX), Ok(usize::MAX)),
}
}
}
crates/clawhdf5-format/src/attribute.rs
+437 -64
View File
@@ -5,8 +5,10 @@ use alloc::{borrow::Cow, string::String, vec::Vec};
#[cfg(feature = "std")]
use std::borrow::Cow;
use crate::addr::to_usize;
use crate::attribute_info::AttributeInfoMessage;
use crate::btree_v2::{BTreeV2Header, collect_btree_v2_records};
use crate::btree_v2::{BTreeV2Header, collect_btree_v2_records_in, find_btree_v2_records_in};
use crate::checksum::jenkins_lookup3;
use crate::data_read;
use crate::dataspace::Dataspace;
use crate::datatype::Datatype;
@@ -15,6 +17,7 @@ use crate::fractal_heap::FractalHeapHeader;
use crate::message_type::MessageType;
use crate::object_header::ObjectHeader;
use crate::shared_message;
use crate::storage::Storage;
use crate::vl_data;
/// A parsed HDF5 attribute message.
@@ -50,7 +53,7 @@ impl AttributeMessage {
///
/// `length_size` is needed for dataspace dimension parsing.
pub fn parse(data: &[u8], length_size: u8) -> Result<AttributeMessage, FormatError> {
Self::parse_impl(data, length_size, None)
Self::parse_impl(data, length_size, None::<(&[u8], u8)>)
}
/// [`AttributeMessage::parse`] with access to the rest of the file, which
@@ -65,13 +68,24 @@ impl AttributeMessage {
offset_size: u8,
length_size: u8,
) -> Result<AttributeMessage, FormatError> {
Self::parse_impl(data, length_size, Some((file_data, offset_size)))
Self::parse_in_storage(data, file_data, offset_size, length_size)
}
fn parse_impl(
/// [`AttributeMessage::parse_in_file`] with the file behind any
/// [`Storage`].
pub fn parse_in_storage<S: Storage + ?Sized>(
data: &[u8],
file: &S,
offset_size: u8,
length_size: u8,
) -> Result<AttributeMessage, FormatError> {
Self::parse_impl(data, length_size, Some((file, offset_size)))
}
fn parse_impl<S: Storage + ?Sized>(
data: &[u8],
length_size: u8,
file: Option<(&[u8], u8)>,
file: Option<(&S, u8)>,
) -> Result<AttributeMessage, FormatError> {
ensure_len(data, 0, 2)?;
let version = data[0];
@@ -86,19 +100,19 @@ impl AttributeMessage {
/// The bytes of an embedded datatype/dataspace message, following the
/// shared-message reference when `shared` is set.
fn embedded_message<'a>(
fn embedded_message<'a, S: Storage + ?Sized>(
bytes: &'a [u8],
shared: bool,
msg_type: MessageType,
length_size: u8,
file: Option<(&[u8], u8)>,
file: Option<(&S, u8)>,
) -> Result<Cow<'a, [u8]>, FormatError> {
if !shared {
return Ok(Cow::Borrowed(bytes));
}
let (file_data, offset_size) = file.ok_or(FormatError::UnresolvedSharedMessage)?;
let shared_ref = shared_message::parse_shared_ref(bytes, offset_size)?;
shared_message::resolve_shared_message(
let shared_ref = shared_message::parse_shared_ref_sized(bytes, offset_size, length_size)?;
shared_message::resolve_shared_message_in(
file_data,
&shared_ref,
msg_type,
@@ -143,10 +157,10 @@ impl AttributeMessage {
})
}
fn parse_v2(
fn parse_v2<S: Storage + ?Sized>(
data: &[u8],
length_size: u8,
file: Option<(&[u8], u8)>,
file: Option<(&S, u8)>,
) -> Result<AttributeMessage, FormatError> {
// Flags: bit 0 = datatype is shared, bit 1 = dataspace is shared.
let flags = data.get(1).copied().unwrap_or(0);
@@ -197,10 +211,10 @@ impl AttributeMessage {
})
}
fn parse_v3(
fn parse_v3<S: Storage + ?Sized>(
data: &[u8],
length_size: u8,
file: Option<(&[u8], u8)>,
file: Option<(&S, u8)>,
) -> Result<AttributeMessage, FormatError> {
// Flags: bit 0 = datatype is shared, bit 1 = dataspace is shared.
let flags = data.get(1).copied().unwrap_or(0);
@@ -322,9 +336,19 @@ impl AttributeMessage {
file_data: &[u8],
offset_size: u8,
length_size: u8,
) -> Result<Vec<String>, FormatError> {
self.read_vl_strings_in(file_data, offset_size, length_size)
}
/// [`Self::read_vl_strings`] over any [`Storage`].
pub fn read_vl_strings_in<S: Storage + ?Sized>(
&self,
file_data: &S,
offset_size: u8,
length_size: u8,
) -> Result<Vec<String>, FormatError> {
let num_elements = self.dataspace.num_elements();
vl_data::read_vl_strings(
vl_data::read_vl_strings_in(
file_data,
&self.raw_data,
num_elements,
@@ -341,7 +365,8 @@ fn compute_raw_data(
dataspace: &Dataspace,
datatype: &Datatype,
) -> Vec<u8> {
let num_elements = dataspace.num_elements() as usize;
// Saturating, like the product: the size is capped at what is there.
let num_elements = usize::try_from(dataspace.num_elements()).unwrap_or(usize::MAX);
let elem_size = datatype.type_size() as usize;
let expected_size = num_elements.saturating_mul(elem_size);
let available = data.len().saturating_sub(pos);
@@ -362,6 +387,18 @@ fn extract_name(bytes: &[u8]) -> String {
String::from_utf8_lossy(&bytes[..end]).into_owned()
}
/// An attribute's datatype gets libhdf5's extra check for a header without
/// a checksum (see [`Datatype::check_unused_bits`]).
fn check_in_header(
attr: AttributeMessage,
header: &ObjectHeader,
) -> Result<AttributeMessage, FormatError> {
if header.version == 1 {
attr.datatype.check_unused_bits()?;
}
Ok(attr)
}
/// Extract all attribute messages from an object header.
pub fn extract_attributes(
header: &ObjectHeader,
@@ -371,7 +408,7 @@ pub fn extract_attributes(
for msg in &header.messages {
if msg.msg_type == MessageType::Attribute {
let attr = AttributeMessage::parse(&msg.data, length_size)?;
attrs.push(attr);
attrs.push(check_in_header(attr, header)?);
}
}
Ok(attrs)
@@ -394,57 +431,302 @@ pub fn find_attribute<'a>(
///
/// Use this instead of `extract_attributes` when reading files that may use dense storage
/// (e.g., objects with many attributes, typically >8).
///
/// Fails if any attribute cannot be read; see [`extract_attributes_tolerant`]
/// to read the others.
pub fn extract_attributes_full(
file_data: &[u8],
header: &ObjectHeader,
offset_size: u8,
length_size: u8,
) -> Result<Vec<AttributeMessage>, FormatError> {
let mut attrs = Vec::new();
extract_attributes_full_in(file_data, header, offset_size, length_size)
}
// Collect compact attributes (inline in OH)
/// [`extract_attributes_full`] over any [`Storage`]. Dense attribute
/// storage is indexed by a v2 B-tree, which is not read over [`Storage`]
/// yet: on a backend without the whole file in memory an object with dense
/// attributes is [`FormatError::ContiguousStorageRequired`].
pub fn extract_attributes_full_in<S: Storage + ?Sized>(
file: &S,
header: &ObjectHeader,
offset_size: u8,
length_size: u8,
) -> Result<Vec<AttributeMessage>, FormatError> {
extract_attributes_with(file, header, offset_size, length_size, &mut Err)
}
/// Like [`extract_attributes_full`], but an attribute that cannot be read
/// (a corrupt or unsupported attribute message, or a heap object that cannot
/// be located) is left out and its error returned alongside the attributes
/// that could be read, instead of failing them all.
///
/// Errors in the structures that index the attributes (the Attribute Info
/// message, the dense-storage heap header or B-tree) still fail the call:
/// then it is unknown which attributes exist at all.
pub fn extract_attributes_tolerant(
file_data: &[u8],
header: &ObjectHeader,
offset_size: u8,
length_size: u8,
) -> Result<(Vec<AttributeMessage>, Vec<FormatError>), FormatError> {
extract_attributes_tolerant_core(file_data, header, offset_size, length_size)
}
/// [`extract_attributes_tolerant`] over any [`Storage`] (see
/// [`extract_attributes_full_in`] for dense storage). One with the whole
/// file in memory is read as the slice, by code compiled in this crate (see
/// [`crate::storage`], "Slice entry points").
#[inline]
pub fn extract_attributes_tolerant_in<S: Storage + ?Sized>(
file_data: &S,
header: &ObjectHeader,
offset_size: u8,
length_size: u8,
) -> Result<(Vec<AttributeMessage>, Vec<FormatError>), FormatError> {
match file_data.as_contiguous() {
Some(all) => extract_attributes_tolerant(all, header, offset_size, length_size),
None => extract_attributes_tolerant_core(file_data, header, offset_size, length_size),
}
}
fn extract_attributes_tolerant_core<S: Storage + ?Sized>(
file_data: &S,
header: &ObjectHeader,
offset_size: u8,
length_size: u8,
) -> Result<(Vec<AttributeMessage>, Vec<FormatError>), FormatError> {
let mut errors = Vec::new();
let attrs = extract_attributes_with(file_data, header, offset_size, length_size, &mut |e| {
errors.push(e);
Ok(())
})?;
Ok((attrs, errors))
}
/// Read every attribute; each one that fails goes to `on_error`, which
/// either stops the read (returns the error) or skips that attribute.
fn extract_attributes_with<S: Storage + ?Sized>(
file_data: &S,
header: &ObjectHeader,
offset_size: u8,
length_size: u8,
on_error: &mut dyn FnMut(FormatError) -> Result<(), FormatError>,
) -> Result<Vec<AttributeMessage>, FormatError> {
let mut attrs = Vec::new();
// Each attribute's creation order, where the file records one.
let mut orders: Vec<u32> = Vec::new();
extract_compact_attributes(
file_data,
header,
offset_size,
length_size,
&mut attrs,
&mut orders,
on_error,
)?;
// Check for dense attributes via AttributeInfo message
let attr_info = find_attribute_info(header, offset_size)?;
if let Some(info) = &attr_info
&& let Some(fh_addr) = info.fractal_heap_address
{
extract_dense_attributes(
file_data,
info,
fh_addr,
offset_size,
length_size,
&mut attrs,
&mut orders,
on_error,
)?;
}
// An object that tracks attribute creation order lists its attributes
// in that order (h5py's `track_order=True`), as libhdf5 does; otherwise
// they come in storage order.
if attr_info.is_some_and(|i| i.max_creation_index.is_some()) {
let mut paired: Vec<(u32, AttributeMessage)> = orders.into_iter().zip(attrs).collect();
paired.sort_by_key(|(o, _)| *o);
attrs = paired.into_iter().map(|(_, a)| a).collect();
}
Ok(attrs)
}
/// B-tree v2 record type of dense attribute storage's name index.
const ATTRIBUTE_NAME_INDEX: u8 = 8;
/// The attribute called `name` on the object with header `header`: the
/// first one [`extract_attributes_tolerant`] returns under that name, or
/// `None` if it returns none (an attribute that cannot be read is not
/// returned there either).
///
/// Compact attributes are in the header and are scanned. Dense attributes
/// are found through the name index (a v2 B-tree of lookup3 name hashes,
/// record type 8): only the attributes whose names hash like `name` are read
/// from the heap, O(log n) instead of all of them. Errors in the structures
/// that index the attributes fail the call, as they fail a listing.
pub fn find_attribute_in_file(
file_data: &[u8],
header: &ObjectHeader,
name: &str,
offset_size: u8,
length_size: u8,
) -> Result<Option<AttributeMessage>, FormatError> {
find_attribute_core(file_data, header, name, offset_size, length_size)
}
/// [`find_attribute_in_file`] over any [`Storage`] (see
/// [`extract_attributes_full_in`] for dense storage, whose name index still
/// needs the whole file in memory). One with the whole file in memory is
/// read as the slice, by code compiled in this crate (see
/// [`crate::storage`], "Slice entry points").
#[inline]
pub fn find_attribute_in<S: Storage + ?Sized>(
file_data: &S,
header: &ObjectHeader,
name: &str,
offset_size: u8,
length_size: u8,
) -> Result<Option<AttributeMessage>, FormatError> {
match file_data.as_contiguous() {
Some(all) => find_attribute_in_file(all, header, name, offset_size, length_size),
None => find_attribute_core(file_data, header, name, offset_size, length_size),
}
}
fn find_attribute_core<S: Storage + ?Sized>(
file_data: &S,
header: &ObjectHeader,
name: &str,
offset_size: u8,
length_size: u8,
) -> Result<Option<AttributeMessage>, FormatError> {
let attr_info = find_attribute_info(header, offset_size)?;
let dense = attr_info
.as_ref()
.and_then(|i| Some((i.fractal_heap_address?, i.btree_name_index_address?)));
let Some((fh_addr, btree_addr)) = dense else {
// Compact only (or dense storage without a name index, which a
// listing reports): as a listing finds it.
return Ok(
extract_attributes_tolerant_in(file_data, header, offset_size, length_size)?
.0
.into_iter()
.find(|a| a.name == name),
);
};
let btree_hdr = BTreeV2Header::parse_in(
file_data,
to_usize(btree_addr)? as u64,
offset_size,
length_size,
)?;
let fh = FractalHeapHeader::parse_in(file_data, fh_addr, offset_size, length_size)?;
if btree_hdr.tree_type != ATTRIBUTE_NAME_INDEX || btree_hdr.record_size < 4 {
return Ok(
extract_attributes_tolerant_in(file_data, header, offset_size, length_size)?
.0
.into_iter()
.find(|a| a.name == name),
);
}
// A listing has the compact attributes first.
let mut compact = Vec::new();
extract_compact_attributes(
file_data,
header,
offset_size,
length_size,
&mut compact,
&mut Vec::new(),
&mut |_| Ok(()),
)?;
if let Some(a) = compact.into_iter().find(|a| a.name == name) {
return Ok(Some(a));
}
// Record: heap ID + message flags(1) + creation order(4) + hash(4); the
// hash is the last field.
let hash = jenkins_lookup3(name.as_bytes());
let hash_at = usize::from(btree_hdr.record_size) - 4;
let records = find_btree_v2_records_in(file_data, &btree_hdr, offset_size, &mut |r| match r
.get(hash_at..hash_at + 4)
{
Some(h) => u32::from_le_bytes([h[0], h[1], h[2], h[3]]).cmp(&hash),
None => core::cmp::Ordering::Less,
})?;
let id_len = usize::from(fh.heap_id_length);
for record in &records {
let Some(id_bytes) = record.data.get(..id_len) else {
continue;
};
let attr = fh
.read_managed_object_in(file_data, id_bytes, offset_size)
.and_then(|d| {
AttributeMessage::parse_in_storage(&d, file_data, offset_size, length_size)
});
// One that cannot be read is left out, as from a listing.
if let Ok(attr) = attr
&& attr.name == name
{
return Ok(Some(attr));
}
}
Ok(None)
}
/// The attributes stored in the object header itself (compact storage), and
/// each one's creation order into `orders`.
fn extract_compact_attributes<S: Storage + ?Sized>(
file_data: &S,
header: &ObjectHeader,
offset_size: u8,
length_size: u8,
attrs: &mut Vec<AttributeMessage>,
orders: &mut Vec<u32>,
on_error: &mut dyn FnMut(FormatError) -> Result<(), FormatError>,
) -> Result<(), FormatError> {
for msg in &header.messages {
if msg.msg_type == MessageType::Attribute {
if shared_message::is_shared(msg.flags) {
let attr = if shared_message::is_shared(msg.flags) {
// Shared attribute: resolve the reference to get actual attribute data
let shared_ref = shared_message::parse_shared_ref(&msg.data, offset_size)?;
let resolved_data = shared_message::resolve_shared_message(
shared_message::parse_shared_ref_sized(&msg.data, offset_size, length_size)
.and_then(|shared_ref| {
shared_message::resolve_shared_message_in(
file_data,
&shared_ref,
MessageType::Attribute,
offset_size,
length_size,
)?;
let attr = AttributeMessage::parse_in_file(
&resolved_data,
)
})
.and_then(|resolved| {
AttributeMessage::parse_in_storage(
&resolved,
file_data,
offset_size,
length_size,
)?;
attrs.push(attr);
)
})
} else {
let attr = AttributeMessage::parse_in_file(
&msg.data,
file_data,
offset_size,
length_size,
)?;
AttributeMessage::parse_in_storage(&msg.data, file_data, offset_size, length_size)
};
let attr = attr.and_then(|a| check_in_header(a, header));
match attr {
Ok(attr) => {
attrs.push(attr);
orders.push(msg.creation_order.map_or(0, u32::from));
}
Err(e) => on_error(e)?,
}
}
}
// Check for dense attributes via AttributeInfo message
let attr_info = find_attribute_info(header, offset_size)?;
if let Some(info) = attr_info
&& let Some(fh_addr) = info.fractal_heap_address
{
let dense_attrs =
extract_dense_attributes(file_data, &info, fh_addr, offset_size, length_size)?;
attrs.extend(dense_attrs);
}
Ok(attrs)
Ok(())
}
/// Find and parse the Attribute Info message from an object header.
@@ -461,16 +743,21 @@ fn find_attribute_info(
Ok(None)
}
/// Extract attributes from dense storage (fractal heap + B-tree v2).
fn extract_dense_attributes(
file_data: &[u8],
/// Extract attributes from dense storage (fractal heap + B-tree v2), and
/// each one's creation order into `orders`.
#[allow(clippy::too_many_arguments)]
fn extract_dense_attributes<S: Storage + ?Sized>(
file_data: &S,
attr_info: &AttributeInfoMessage,
fh_addr: u64,
offset_size: u8,
length_size: u8,
) -> Result<Vec<AttributeMessage>, FormatError> {
attrs: &mut Vec<AttributeMessage>,
orders: &mut Vec<u32>,
on_error: &mut dyn FnMut(FormatError) -> Result<(), FormatError>,
) -> Result<(), FormatError> {
// Parse fractal heap
let fh = FractalHeapHeader::parse(file_data, fh_addr as usize, offset_size, length_size)?;
let fh = FractalHeapHeader::parse_in(file_data, fh_addr, offset_size, length_size)?;
// Parse B-tree v2 for name index (type 8)
let btree_addr = attr_info
@@ -479,31 +766,47 @@ fn extract_dense_attributes(
expected: 1,
available: 0,
})?;
let btree_hdr = BTreeV2Header::parse(file_data, btree_addr as usize, offset_size, length_size)?;
let records = collect_btree_v2_records(file_data, &btree_hdr, offset_size, length_size)?;
let btree_hdr = BTreeV2Header::parse_in(
file_data,
to_usize(btree_addr)? as u64,
offset_size,
length_size,
)?;
let records = collect_btree_v2_records_in(file_data, &btree_hdr, offset_size, length_size)?;
let mut attrs = Vec::new();
for record in &records {
// Per HDF5 spec, both type 8 and type 9 records start with heap_id:
// Type 8: heap_id(8) + msg_flags(1) + creation_order(4) + hash(4)
// Type 9: heap_id(8) + msg_flags(1) + creation_order(4)
let id_offset = 0;
if record.data.len() < id_offset + fh.heap_id_length as usize {
let id_len = fh.heap_id_length as usize;
let Some(id_bytes) = record.data.get(..id_len) else {
on_error(FormatError::UnexpectedEof {
expected: id_len,
available: record.data.len(),
})?;
continue;
}
let id_bytes = &record.data[id_offset..id_offset + fh.heap_id_length as usize];
// Read attribute message from fractal heap
let attr_data = fh.read_managed_object(file_data, id_bytes, offset_size)?;
};
// The data in the heap is a complete attribute message
let attr =
AttributeMessage::parse_in_file(&attr_data, file_data, offset_size, length_size)?;
let attr = fh
.read_managed_object_in(file_data, id_bytes, offset_size)
.and_then(|attr_data| {
AttributeMessage::parse_in_storage(&attr_data, file_data, offset_size, length_size)
});
match attr {
Ok(attr) => {
attrs.push(attr);
let order = record
.data
.get(id_len + 1..id_len + 5)
.map_or(0, |b| u32::from_le_bytes([b[0], b[1], b[2], b[3]]));
orders.push(order);
}
Err(e) => on_error(e)?,
}
}
Ok(attrs)
Ok(())
}
#[cfg(test)]
@@ -523,7 +826,8 @@ mod tests {
/// Build an f64 LE datatype message.
fn build_f64_dt() -> Vec<u8> {
let mut buf = build_dt_header(1, 1, [0x00, 0x00, 0x02], 8);
// Sign bit 63 (bits 8-15 of the class bits).
let mut buf = build_dt_header(1, 1, [0x20, 63, 0x00], 8);
let mut props = [0u8; 12];
props[2..4].copy_from_slice(&64u16.to_le_bytes()); // bit_precision
props[4] = 52; // exp_location
@@ -897,4 +1201,73 @@ mod tests {
let strs = attr.read_as_strings().unwrap();
assert_eq!(strs, vec!["abcd", "EFGH"]);
}
/// Every object's attributes in h5py-written files read identically
/// through a read_at-only CountingStorage — compact ones, shared ones,
/// those behind an Attribute Info message and dense storage (its v2
/// B-tree name index included) — and through a slice as Storage.
#[test]
fn storage_reads_match_slice_reads() {
use crate::storage::CountingStorage;
let files: [(&str, &[u8]); 5] = [
("attrs", include_bytes!("../tests/fixtures/attrs.h5")),
(
"mixed_attrs",
include_bytes!("../tests/fixtures/mixed_attrs.h5"),
),
(
"dense_attrs",
include_bytes!("../tests/fixtures/dense_attrs.h5"),
),
(
"dense_attrs_root",
include_bytes!("../tests/fixtures/dense_attrs_root.h5"),
),
(
"shared_fill_value",
include_bytes!("../tests/fixtures/shared_fill_value.h5"),
),
];
let (mut same, mut dense, mut attrs) = (0, 0, 0);
for (name, file) in files {
let sb = crate::superblock::Superblock::parse(file, 0).unwrap();
let (os, ls) = (sb.offset_size, sb.length_size);
let mut addrs = vec![sb.root_group_address];
addrs.extend(
crate::group_v2::resolve_group_children(file, &sb, sb.root_group_address)
.unwrap()
.iter()
.map(|e| e.object_header_address),
);
let storage = CountingStorage::new(file.to_vec());
for addr in addrs {
let header = ObjectHeader::parse(file, addr as usize, os, ls).unwrap();
let want = extract_attributes_full(file, &header, os, ls);
let slice_storage = extract_attributes_full_in(&file, &header, os, ls);
assert_eq!(format!("{slice_storage:?}"), format!("{want:?}"));
let got = extract_attributes_full_in(&storage, &header, os, ls);
let got_t = extract_attributes_tolerant_in(&storage, &header, os, ls);
let is_dense = find_attribute_info(&header, os)
.unwrap()
.is_some_and(|i| i.fractal_heap_address.is_some());
if is_dense {
dense += 1;
}
attrs += want.as_ref().map_or(0, Vec::len);
assert_eq!(format!("{got:?}"), format!("{want:?}"), "{name}");
let want_t = extract_attributes_tolerant(file, &header, os, ls);
assert_eq!(format!("{got_t:?}"), format!("{want_t:?}"), "{name}");
same += 1;
for a in want.iter().flatten() {
let one = find_attribute_in(&storage, &header, &a.name, os, ls);
let want_one = find_attribute_in_file(file, &header, &a.name, os, ls);
assert_eq!(format!("{one:?}"), format!("{want_one:?}"), "{name}");
}
}
}
assert!(
same >= 5 && dense >= 2 && attrs >= 5,
"{same} {dense} {attrs}"
);
}
}
crates/clawhdf5-format/src/btree_v1.rs
+89 -20
View File
@@ -4,6 +4,7 @@
use alloc::vec::Vec;
use crate::error::FormatError;
use crate::storage::{Storage, read_exact_at};
/// A parsed B-tree v1 node.
#[derive(Debug, Clone)]
@@ -74,13 +75,28 @@ impl BTreeV1Node {
file_data: &[u8],
offset: usize,
offset_size: u8,
length_size: u8,
) -> Result<BTreeV1Node, FormatError> {
Self::parse_in(file_data, offset as u64, offset_size, length_size)
}
/// [`Self::parse`] over any [`Storage`]: one read of the node's header,
/// one of its keys and children.
pub fn parse_in<S: Storage + ?Sized>(
file: &S,
offset: u64,
offset_size: u8,
_length_size: u8,
) -> Result<BTreeV1Node, FormatError> {
// signature(4) + node_type(1) + node_level(1) + entries_used(2) = 8
// + left_sibling(offset_size) + right_sibling(offset_size)
let os = offset_size as usize;
let header_size = 8 + os * 2;
ensure_len(file_data, offset, header_size)?;
let header = read_exact_at(file, offset, header_size)?;
let file_data: &[u8] = &header;
// The header's read checked that `offset + header_size` fits.
let body_start = offset + header_size as u64;
let offset = 0usize;
if &file_data[offset..offset + 4] != b"TREE" {
return Err(FormatError::InvalidBTreeSignature);
@@ -102,31 +118,30 @@ impl BTreeV1Node {
} else {
Some(read_offset(file_data, pos, offset_size)?)
};
pos += os;
// For type 0: keys are offset_size bytes, children are offset_size bytes
// Layout: key[0], child[0], key[1], child[1], ..., key[N-1], child[N-1], key[N]
let eu = entries_used as usize;
let key_size = os; // For type 0, key = offset_size
let needed = eu * (key_size + os) + key_size; // eu children + (eu+1) keys
ensure_len(file_data, pos, needed)?;
let body = read_exact_at(file, body_start, needed)?;
let file_data: &[u8] = &body;
let mut keys = Vec::with_capacity(eu + 1);
let mut children = Vec::with_capacity(eu);
for _i in 0..eu {
// key[i]
let key = read_offset(file_data, pos, offset_size)?;
keys.push(key);
pos += key_size;
// child[i]
let child = read_offset(file_data, pos, offset_size)?;
children.push(child);
pos += os;
if os == 0 {
// What reading the first key reports (and keeps `chunks_exact`
// below from being given a zero size).
return Err(FormatError::InvalidOffsetSize(offset_size));
}
// final key
let key = read_offset(file_data, pos, offset_size)?;
keys.push(key);
// `needed` bytes: key[0], child[0], ..., child[eu - 1], key[eu].
let (pairs, last) = file_data.split_at(eu * (key_size + os));
for pair in pairs.chunks_exact(key_size + os) {
keys.push(read_offset(pair, 0, offset_size)?);
children.push(read_offset(pair, key_size, offset_size)?);
}
keys.push(read_offset(last, 0, offset_size)?);
Ok(BTreeV1Node {
node_type,
@@ -150,11 +165,21 @@ pub fn collect_symbol_table_nodes(
offset_size: u8,
length_size: u8,
) -> Result<Vec<u64>, FormatError> {
collect_symbol_table_nodes_inner(file_data, btree_address, offset_size, length_size, 0)
collect_symbol_table_nodes_in(file_data, btree_address, offset_size, length_size)
}
fn collect_symbol_table_nodes_inner(
file_data: &[u8],
/// [`collect_symbol_table_nodes`] over any [`Storage`]: two reads per node.
pub fn collect_symbol_table_nodes_in<S: Storage + ?Sized>(
file: &S,
btree_address: u64,
offset_size: u8,
length_size: u8,
) -> Result<Vec<u64>, FormatError> {
collect_symbol_table_nodes_inner(file, btree_address, offset_size, length_size, 0)
}
fn collect_symbol_table_nodes_inner<S: Storage + ?Sized>(
file: &S,
btree_address: u64,
offset_size: u8,
length_size: u8,
@@ -164,7 +189,7 @@ fn collect_symbol_table_nodes_inner(
return Err(FormatError::NestingDepthExceeded);
}
let node = BTreeV1Node::parse(file_data, btree_address as usize, offset_size, length_size)?;
let node = BTreeV1Node::parse_in(file, btree_address, offset_size, length_size)?;
if node.node_type != 0 {
return Err(FormatError::InvalidBTreeNodeType(node.node_type));
@@ -178,7 +203,7 @@ fn collect_symbol_table_nodes_inner(
let mut result = Vec::new();
for &child_addr in &node.children {
let child_snods = collect_symbol_table_nodes_inner(
file_data,
file,
child_addr,
offset_size,
length_size,
@@ -317,4 +342,48 @@ mod tests {
assert_eq!(node.entries_used, 1);
assert_eq!(node.children, vec![0x50]);
}
/// Nodes and trees, cut at every length, parse identically through a
/// `read_at`-only storage.
#[test]
fn storage_parse_matches_slice_parse() {
use crate::storage::CountingStorage;
let nodes = [
build_btree_node(0, 0, &[0, 5, 10], &[0x100, 0x200], None, None, 8),
build_btree_node(0, 0, &[0, 5], &[0x100], Some(0x40), Some(0x80), 4),
build_btree_node(1, 2, &[0, 5], &[0x100], None, Some(0x80), 8),
];
for (n, node) in nodes.iter().enumerate() {
let os = if n == 1 { 4 } else { 8 };
for cut in 0..=node.len() {
let f = &node[..cut];
let storage = CountingStorage::new(f.to_vec());
let want = BTreeV1Node::parse(f, 0, os, 8);
let got = BTreeV1Node::parse_in(&storage, 0, os, 8);
assert_eq!(format!("{got:?}"), format!("{want:?}"));
}
}
let leaf1 = build_btree_node(0, 0, &[0, 5], &[0xA00], None, None, 8);
let leaf2 = build_btree_node(0, 0, &[5, 10], &[0xB00], None, None, 8);
let internal = build_btree_node(0, 1, &[0, 5, 10], &[0, 256], None, None, 8);
let mut file = vec![0u8; 512 + internal.len()];
file[..leaf1.len()].copy_from_slice(&leaf1);
file[256..256 + leaf2.len()].copy_from_slice(&leaf2);
file[512..].copy_from_slice(&internal);
for cut in [file.len(), 300, 260, 100, 10] {
let mut f = file.clone();
if cut < 512 {
// Truncate the leaves, keep the root.
f[cut..512].fill(0);
}
let storage = CountingStorage::new(f.clone());
assert_eq!(
collect_symbol_table_nodes_in(&storage, 512, 8, 8),
collect_symbol_table_nodes(&f, 512, 8, 8)
);
}
let storage = CountingStorage::new(file);
collect_symbol_table_nodes_in(&storage, 512, 8, 8).unwrap();
assert_eq!(storage.reads(), 6);
}
}
crates/clawhdf5-format/src/btree_v2.rs
+635 -148
View File
@@ -2,11 +2,14 @@
#[cfg(not(feature = "std"))]
use alloc::vec::Vec;
use core::cmp::Ordering;
#[cfg(feature = "checksum")]
use byteorder::{ByteOrder, LittleEndian};
use crate::addr::to_usize;
use crate::error::FormatError;
use crate::storage::{Storage, Window, len_usize};
/// Parsed B-tree v2 header (signature "BTHD").
#[derive(Debug, Clone)]
@@ -71,7 +74,7 @@ fn ensure_len(data: &[u8], pos: usize, needed: usize) -> Result<(), FormatError>
/// Compute the number of bytes needed to represent a count, using variable-width encoding.
/// B-tree v2 uses this for the number of records fields in internal nodes.
fn bytes_for_max_records(max_nrec: u64) -> usize {
pub(crate) fn bytes_for_max_records(max_nrec: u64) -> usize {
if max_nrec == 0 {
return 1;
}
@@ -97,38 +100,52 @@ impl BTreeV2Header {
offset_size: u8,
length_size: u8,
) -> Result<BTreeV2Header, FormatError> {
ensure_len(file_data, offset, 4)?;
if &file_data[offset..offset + 4] != b"BTHD" {
Self::parse_in(file_data, offset as u64, offset_size, length_size)
}
/// [`Self::parse`] over any [`Storage`]: one bounded read of the
/// header.
pub fn parse_in<S: Storage + ?Sized>(
file: &S,
offset: u64,
offset_size: u8,
length_size: u8,
) -> Result<BTreeV2Header, FormatError> {
// Every field and the checksum; the window holds all of it or ends
// at the end of the file, so its bounds checks are the whole-file
// ones.
let full = 16 + usize::from(offset_size) + 2 + usize::from(length_size) + 4;
let w = Window::read(file, offset, full)?;
let d = &w.bytes;
w.ensure(0, 4)?;
if &d[..4] != b"BTHD" {
return Err(FormatError::InvalidBTreeV2Signature);
}
ensure_len(file_data, offset, 4 + 1 + 1 + 4 + 2 + 2 + 1 + 1)?;
let version = file_data[offset + 4];
w.ensure(0, 4 + 1 + 1 + 4 + 2 + 2 + 1 + 1)?;
let version = d[4];
if version != 0 {
return Err(FormatError::InvalidBTreeV2Version(version));
}
let tree_type = file_data[offset + 5];
let node_size = u32::from_le_bytes([
file_data[offset + 6],
file_data[offset + 7],
file_data[offset + 8],
file_data[offset + 9],
]);
let record_size = u16::from_le_bytes([file_data[offset + 10], file_data[offset + 11]]);
let depth = u16::from_le_bytes([file_data[offset + 12], file_data[offset + 13]]);
let _split_percent = file_data[offset + 14];
let _merge_percent = file_data[offset + 15];
let tree_type = d[5];
let node_size = u32::from_le_bytes([d[6], d[7], d[8], d[9]]);
let record_size = u16::from_le_bytes([d[10], d[11]]);
let depth = u16::from_le_bytes([d[12], d[13]]);
let _split_percent = d[14];
let _merge_percent = d[15];
let mut pos = offset + 16;
let root_node_address = read_offset(file_data, pos, offset_size)?;
let mut pos = 16;
w.ensure(pos, usize::from(offset_size))?;
let root_node_address = read_offset(d, pos, offset_size)?;
pos += offset_size as usize;
ensure_len(file_data, pos, 2)?;
let num_records_in_root = u16::from_le_bytes([file_data[pos], file_data[pos + 1]]);
w.ensure(pos, 2)?;
let num_records_in_root = u16::from_le_bytes([d[pos], d[pos + 1]]);
pos += 2;
let total_records = read_offset(file_data, pos, length_size)?;
w.ensure(pos, usize::from(length_size))?;
let total_records = read_offset(d, pos, length_size)?;
#[allow(unused_assignments)]
{
pos += length_size as usize;
@@ -137,9 +154,9 @@ impl BTreeV2Header {
// Validate header checksum
#[cfg(feature = "checksum")]
{
ensure_len(file_data, pos, 4)?;
let stored = LittleEndian::read_u32(&file_data[pos..pos + 4]);
let computed = crate::checksum::jenkins_lookup3(&file_data[offset..pos]);
w.ensure(pos, 4)?;
let stored = LittleEndian::read_u32(&d[pos..pos + 4]);
let computed = crate::checksum::jenkins_lookup3(&d[..pos]);
if computed != stored {
return Err(FormatError::ChecksumMismatch {
expected: stored,
@@ -163,7 +180,7 @@ impl BTreeV2Header {
/// Compute maximum records per node for a given depth level.
/// leaf: (node_size - overhead) / record_size
/// internal: depends on pointers
fn max_records_leaf(node_size: u32, record_size: u16) -> u64 {
pub(crate) fn max_records_leaf(node_size: u32, record_size: u16) -> u64 {
// Leaf overhead: signature(4) + version(1) + type(1) + checksum(4) = 10
let overhead = 10u32;
if node_size <= overhead || record_size == 0 {
@@ -172,33 +189,72 @@ fn max_records_leaf(node_size: u32, record_size: u16) -> u64 {
((node_size - overhead) / record_size as u32) as u64
}
/// Deepest B-tree v2 accepted. See [`collect_btree_v2_records`].
const MAX_DEPTH: u16 = 64;
/// Take `n` records from the traversal's budget, or refuse the tree.
fn spend(budget: &mut usize, n: usize) -> Result<(), FormatError> {
*budget = budget
.checked_sub(n)
.ok_or(FormatError::NestingDepthExceeded)?;
Ok(())
}
/// Collect all records from a B-tree v2 by traversing from the root.
pub fn collect_btree_v2_records(
file_data: &[u8],
header: &BTreeV2Header,
offset_size: u8,
length_size: u8,
) -> Result<Vec<BTreeV2Record>, FormatError> {
collect_btree_v2_records_in(file_data, header, offset_size, length_size)
}
/// [`collect_btree_v2_records`] over any [`Storage`]: one bounded read per
/// node.
pub fn collect_btree_v2_records_in<S: Storage + ?Sized>(
file: &S,
header: &BTreeV2Header,
offset_size: u8,
length_size: u8,
) -> Result<Vec<BTreeV2Record>, FormatError> {
if header.total_records == 0 || header.num_records_in_root == 0 {
return Ok(Vec::new());
}
// Recursion is one frame per level, and the depth is read from the file:
// a crafted header claiming 65 535 levels over a node that is its own
// child overflowed the stack. 64 matches the fractal heap's guard, and no
// real tree comes close — even at the minimum fan-out of two it would
// hold more than 2^64 records.
if header.depth > MAX_DEPTH {
return Err(FormatError::NestingDepthExceeded);
}
// A valid tree stores each record once, in its own bytes, so it cannot
// hold more records than the file has room for. Children are addresses,
// though, and nothing makes them distinct: levels whose children all
// point at one shared node below reach it fan-out^depth times, which is
// millions of records from a few kilobytes. Counting against what the
// file could physically contain bounds that without trusting the
// header's own `total_records`.
let mut budget = len_usize(file) / usize::from(header.record_size.max(1));
let max_leaf_nrec = max_records_leaf(header.node_size, header.record_size);
if header.depth == 0 {
// Root is a leaf
parse_leaf_records(
file_data,
header.root_node_address as usize,
file,
to_usize(header.root_node_address)?,
header.num_records_in_root,
header.record_size,
header.node_size,
)
} else {
// Root is internal; traverse recursively
let mut records = Vec::new();
collect_internal_records(
file_data,
header.root_node_address as usize,
file,
to_usize(header.root_node_address)?,
header.num_records_in_root,
header.depth,
header.record_size,
@@ -206,42 +262,79 @@ pub fn collect_btree_v2_records(
offset_size,
length_size,
max_leaf_nrec,
&mut budget,
&mut records,
)?;
Ok(records)
}
}
/// A node's bytes: `want` bytes at `offset` (fewer only at the end of the
/// file), after checking its 4-byte signature. A node is read in one piece
/// when it fits in `node_size` (every valid node does); a larger claimed
/// extent — record counts from a damaged parent — is first checked against
/// the end of the file, so it costs a read only of bytes the file has.
/// Bounds errors are the whole-file ones: the signature check needs the
/// first 6 bytes, then `checks` — `(position, length)` pairs relative to
/// the node, in the order the parser checks them — must lie in the file.
fn read_node<'a, S: Storage + ?Sized>(
file: &'a S,
offset: usize,
want: usize,
node_size: u32,
signature: &[u8; 4],
checks: &[(usize, usize)],
) -> Result<Window<'a>, FormatError> {
let one_read = usize::try_from(node_size).unwrap_or(usize::MAX).max(6);
let w = Window::read(file, offset as u64, want.min(one_read))?;
w.ensure(0, 6)?;
if &w.bytes[..4] != signature {
return Err(FormatError::InvalidBTreeV2Signature);
}
if want <= one_read {
return Ok(w);
}
for &(rel, len) in checks {
Window::check_extent(file, offset as u64, rel, len)?;
}
Window::read(file, offset as u64, want)
}
/// Parse records from a leaf node (signature "BTLF").
fn parse_leaf_records(
file_data: &[u8],
fn parse_leaf_records<S: Storage + ?Sized>(
file: &S,
offset: usize,
num_records: u16,
record_size: u16,
node_size: u32,
) -> Result<Vec<BTreeV2Record>, FormatError> {
// signature(4) + version(1) + type(1) = 6 bytes header
ensure_len(file_data, offset, 6)?;
if &file_data[offset..offset + 4] != b"BTLF" {
return Err(FormatError::InvalidBTreeV2Signature);
}
let pos = offset + 6;
let pos = 6;
let rs = record_size as usize;
let total = (num_records as usize)
.checked_mul(rs)
.ok_or(FormatError::UnexpectedEof {
expected: usize::MAX,
available: file_data.len(),
available: len_usize(file),
})?;
ensure_len(file_data, pos, total)?;
let w = read_node(
file,
offset,
pos + total + 4,
node_size,
b"BTLF",
&[(pos, total)],
)?;
let d = &w.bytes;
w.ensure(pos, total)?;
// Validate checksum: 4 bytes after records + padding
#[cfg(feature = "checksum")]
{
let checksum_pos = pos + total;
if file_data.len() >= checksum_pos + 4 {
let stored = LittleEndian::read_u32(&file_data[checksum_pos..checksum_pos + 4]);
let computed = crate::checksum::jenkins_lookup3(&file_data[offset..checksum_pos]);
if d.len() >= checksum_pos + 4 {
let stored = LittleEndian::read_u32(&d[checksum_pos..checksum_pos + 4]);
let computed = crate::checksum::jenkins_lookup3(&d[..checksum_pos]);
if computed != stored {
return Err(FormatError::ChecksumMismatch {
expected: stored,
@@ -255,16 +348,135 @@ fn parse_leaf_records(
for i in 0..num_records as usize {
let start = pos + i * rs;
records.push(BTreeV2Record {
data: file_data[start..start + rs].to_vec(),
data: d[start..start + rs].to_vec(),
});
}
Ok(records)
}
/// An internal node read from the file: its bytes (from the signature on),
/// where its records start, and its children as `(address, record count)`.
struct InternalNode<'a> {
node: Window<'a>,
records_start: usize,
children: Vec<(u64, u16)>,
}
impl InternalNode<'_> {
/// Record `i`, `rs` bytes long.
fn record(&self, i: usize, rs: usize) -> Result<&[u8], FormatError> {
let overflow = || FormatError::UnexpectedEof {
expected: usize::MAX,
available: usize::MAX,
};
let rec_start = i
.checked_mul(rs)
.and_then(|o| self.records_start.checked_add(o))
.ok_or_else(overflow)?;
self.node.ensure(rec_start, rs)?;
Ok(&self.node.bytes[rec_start..rec_start + rs])
}
}
/// An internal node's layout: where its records start, and its children as
/// `(address, record count)`.
#[allow(clippy::too_many_arguments)]
fn read_internal_node<S: Storage + ?Sized>(
file: &S,
offset: usize,
num_records: u16,
depth: u16,
record_size: u16,
node_size: u32,
offset_size: u8,
max_leaf_nrec: u64,
) -> Result<InternalNode<'_>, FormatError> {
let nr = num_records as usize;
let rs = record_size as usize;
// Records first
let records_total = nr.checked_mul(rs).ok_or(FormatError::UnexpectedEof {
expected: usize::MAX,
available: len_usize(file),
})?;
// Child pointer layout, as libhdf5 computes it (H5B2__hdr_init): the
// child's record count is always encoded in the width needed for a
// *leaf's* maximum, and — below the first internal level — the child
// subtree's total record count in the width needed for the most records
// a subtree of that depth can hold.
let child_depth = depth - 1;
let nrec_width = bytes_for_max_records(max_leaf_nrec);
let total_nrec_width = if depth > 1 {
bytes_for_max_records(cum_max_records(
node_size,
record_size,
offset_size,
max_leaf_nrec,
child_depth,
))
} else {
0
};
let num_children = nr + 1;
let child_ptr_size = offset_size as usize + nrec_width + total_nrec_width;
let pointers = num_children * child_ptr_size;
// signature(4) + version(1) + type(1) = 6, records, pointers, checksum.
let w = read_node(
file,
offset,
6 + records_total + pointers + 4,
node_size,
b"BTIN",
&[(6, records_total), (6 + records_total, pointers)],
)?;
let d = &w.bytes;
let mut pos = 6;
w.ensure(pos, records_total)?;
let records_start = pos;
pos += records_total;
w.ensure(pos, pointers)?;
let mut children = Vec::with_capacity(num_children);
for _ in 0..num_children {
let addr = read_offset(d, pos, offset_size)?;
pos += offset_size as usize;
let child_nrec = read_var_uint(d, pos, nrec_width)? as u16;
pos += nrec_width;
pos += total_nrec_width; // skip total records in subtree
children.push((addr, child_nrec));
}
// The checksum follows the child pointers and covers the node up to it.
// Lookups prune children by the keys in this node, so an unverified
// internal node could hide a record without any error: libhdf5 refuses
// a mismatch here, and so does this.
#[cfg(feature = "checksum")]
{
w.ensure(pos, 4)?;
let stored = LittleEndian::read_u32(&d[pos..pos + 4]);
let computed = crate::checksum::jenkins_lookup3(&d[..pos]);
if computed != stored {
return Err(FormatError::ChecksumMismatch {
expected: stored,
computed,
});
}
}
Ok(InternalNode {
node: w,
records_start,
children,
})
}
/// Recursively collect records from an internal node.
#[allow(clippy::too_many_arguments, clippy::only_used_in_recursion)]
fn collect_internal_records(
file_data: &[u8],
fn collect_internal_records<S: Storage + ?Sized>(
file: &S,
offset: usize,
num_records: u16,
depth: u16,
@@ -273,90 +485,41 @@ fn collect_internal_records(
offset_size: u8,
length_size: u8,
max_leaf_nrec: u64,
budget: &mut usize,
out: &mut Vec<BTreeV2Record>,
) -> Result<(), FormatError> {
// signature(4) + version(1) + type(1) = 6
ensure_len(file_data, offset, 6)?;
if &file_data[offset..offset + 4] != b"BTIN" {
return Err(FormatError::InvalidBTreeV2Signature);
}
let nr = num_records as usize;
let rs = record_size as usize;
let mut pos = offset + 6;
// Read all records first
let records_total = nr.checked_mul(rs).ok_or(FormatError::UnexpectedEof {
expected: usize::MAX,
available: file_data.len(),
})?;
ensure_len(file_data, pos, records_total)?;
let records_start = pos;
pos += records_total;
// Compute sizes for child pointers
// max_records at child depth - for variable-width nrec encoding
let node = read_internal_node(
file,
offset,
num_records,
depth,
record_size,
node_size,
offset_size,
max_leaf_nrec,
)?;
let child_depth = depth - 1;
let max_nrec_child = if child_depth == 0 {
max_leaf_nrec
} else {
// For internal nodes at child_depth, the true max_nrec depends on the
// node size, record size, and the recursive width of child pointer
// entries (which themselves depend on max_nrec at deeper levels).
// Computing the exact value requires iterating from the leaf level
// upward, as described in the HDF5 spec (III.A.2 "Computing the Size
// of B-tree Nodes").
//
// We use `max_leaf_nrec * 2` as a conservative upper bound. This
// over-estimates the nrec encoding width, which means we may read
// slightly more bytes per child pointer than strictly necessary, but
// never fewer. The over-read bytes are harmless because we only
// decode `num_records` entries (the actual count from the node header).
//
// Known limitation: for very deep trees (depth > 3) with small record
// sizes, the true max could exceed this estimate, causing us to
// under-allocate the nrec encoding width and misparse child pointers.
// In practice, HDF5 B-tree v2 depths rarely exceed 2-3.
max_leaf_nrec * 2
};
let nrec_width = bytes_for_max_records(max_nrec_child);
// Total records in subtree width (only if depth > 1)
let total_nrec_width = if depth > 1 {
// Width to hold total records in a subtree
// We compute max possible total records at this subtree depth
let max_total = header_max_total_records(max_leaf_nrec, depth - 1);
bytes_for_max_records(max_total)
} else {
0
};
let num_children = nr + 1;
let child_ptr_size = offset_size as usize + nrec_width + total_nrec_width;
ensure_len(file_data, pos, num_children * child_ptr_size)?;
// Read child pointers
let mut children = Vec::with_capacity(num_children);
for _ in 0..num_children {
let addr = read_offset(file_data, pos, offset_size)?;
pos += offset_size as usize;
let child_nrec = read_var_uint(file_data, pos, nrec_width)? as u16;
pos += nrec_width;
pos += total_nrec_width; // skip total records in subtree
children.push((addr, child_nrec));
}
// Interleave: child[0], record[0], child[1], record[1], ..., child[nr]
// We collect child[0] records, then record[0], then child[1], etc.
for (i, &(child_addr, child_nrec)) in children.iter().enumerate() {
for (i, &(child_addr, child_nrec)) in node.children.iter().enumerate() {
if child_depth == 0 {
let leaf_recs =
parse_leaf_records(file_data, child_addr as usize, child_nrec, record_size)?;
// Before parsing, so a refused tree is not also a large allocation.
spend(budget, usize::from(child_nrec))?;
let leaf_recs = parse_leaf_records(
file,
to_usize(child_addr)?,
child_nrec,
record_size,
node_size,
)?;
out.extend(leaf_recs);
} else {
collect_internal_records(
file_data,
child_addr as usize,
file,
to_usize(child_addr)?,
child_nrec,
child_depth,
record_size,
@@ -364,37 +527,17 @@ fn collect_internal_records(
offset_size,
length_size,
max_leaf_nrec,
budget,
out,
)?;
}
// Add record[i] (except after the last child)
if i < nr {
let rec_offset = i.checked_mul(rs).ok_or(FormatError::UnexpectedEof {
expected: usize::MAX,
available: file_data.len(),
})?;
let rec_start =
records_start
.checked_add(rec_offset)
.ok_or(FormatError::UnexpectedEof {
expected: usize::MAX,
available: file_data.len(),
})?;
let rec_end = rec_start
.checked_add(rs)
.ok_or(FormatError::UnexpectedEof {
expected: usize::MAX,
available: file_data.len(),
})?;
if rec_end > file_data.len() {
return Err(FormatError::UnexpectedEof {
expected: rec_end,
available: file_data.len(),
});
}
let data = node.record(i, rs)?;
spend(budget, 1)?;
out.push(BTreeV2Record {
data: file_data[rec_start..rec_end].to_vec(),
data: data.to_vec(),
});
}
}
@@ -402,14 +545,218 @@ fn collect_internal_records(
Ok(())
}
/// Estimate maximum total records at a given depth (for variable-width encoding).
fn header_max_total_records(max_leaf_nrec: u64, depth: u16) -> u64 {
// Conservative: branching factor * max_leaf at each level
let mut total = max_leaf_nrec;
for _ in 0..depth {
total = total.saturating_mul(max_leaf_nrec.max(2));
/// The records of a B-tree v2 that fall in one key range, found by
/// descending the tree instead of reading all of it.
///
/// `cmp` places a record relative to the range: `Less` if the record sorts
/// before it, `Greater` if after, `Equal` if the record is in it. The tree
/// must be ordered consistently with `cmp`, as libhdf5 orders it (a link or
/// attribute name index by name hash, so all records with one hash form a
/// range whatever order their names are in). Only the nodes whose key
/// interval overlaps the range are read: O(depth) nodes plus those holding
/// the matches. Matches come in tree order.
pub fn find_btree_v2_records(
file_data: &[u8],
header: &BTreeV2Header,
offset_size: u8,
cmp: &mut dyn FnMut(&[u8]) -> Ordering,
) -> Result<Vec<BTreeV2Record>, FormatError> {
find_btree_v2_records_in(file_data, header, offset_size, cmp)
}
total
/// [`find_btree_v2_records`] over any [`Storage`]: one bounded read per
/// node visited.
pub fn find_btree_v2_records_in<S: Storage + ?Sized>(
file: &S,
header: &BTreeV2Header,
offset_size: u8,
cmp: &mut dyn FnMut(&[u8]) -> Ordering,
) -> Result<Vec<BTreeV2Record>, FormatError> {
if header.total_records == 0 || header.num_records_in_root == 0 {
return Ok(Vec::new());
}
if header.depth > MAX_DEPTH {
return Err(FormatError::NestingDepthExceeded);
}
// As in `collect_btree_v2_records`: a valid tree cannot hold more
// records than the file has room for, however its children are shared.
let mut budget = len_usize(file) / usize::from(header.record_size.max(1));
let max_leaf_nrec = max_records_leaf(header.node_size, header.record_size);
let mut out = Vec::new();
find_in_node(
file,
header,
to_usize(header.root_node_address)?,
header.num_records_in_root,
header.depth,
offset_size,
max_leaf_nrec,
cmp,
&mut budget,
&mut out,
)?;
Ok(out)
}
#[allow(clippy::too_many_arguments)]
fn find_in_node<S: Storage + ?Sized>(
file: &S,
header: &BTreeV2Header,
offset: usize,
num_records: u16,
depth: u16,
offset_size: u8,
max_leaf_nrec: u64,
cmp: &mut dyn FnMut(&[u8]) -> Ordering,
budget: &mut usize,
out: &mut Vec<BTreeV2Record>,
) -> Result<(), FormatError> {
spend(budget, usize::from(num_records))?;
if depth == 0 {
let records = parse_leaf_records(
file,
offset,
num_records,
header.record_size,
header.node_size,
)?;
out.extend(
records
.into_iter()
.filter(|r| cmp(&r.data) == Ordering::Equal),
);
return Ok(());
}
let rs = usize::from(header.record_size);
let node = read_internal_node(
file,
offset,
num_records,
depth,
header.record_size,
header.node_size,
offset_size,
max_leaf_nrec,
)?;
let nr = usize::from(num_records);
let mut order = Vec::with_capacity(nr);
for i in 0..nr {
order.push(cmp(node.record(i, rs)?));
}
// Child `i` holds the keys between record `i - 1` and record `i`: it can
// hold a match unless the record before it is already past the range or
// the record after it is still before it.
for (i, &(child_addr, child_nrec)) in node.children.iter().enumerate() {
let after_left = i == 0 || order[i - 1] != Ordering::Greater;
let before_right = i == nr || order[i] != Ordering::Less;
if after_left && before_right {
find_in_node(
file,
header,
to_usize(child_addr)?,
child_nrec,
depth - 1,
offset_size,
max_leaf_nrec,
cmp,
budget,
out,
)?;
}
if i < nr && order[i] == Ordering::Equal {
out.push(BTreeV2Record {
data: node.record(i, rs)?.to_vec(),
});
}
}
Ok(())
}
/// Most records a subtree whose root is at `depth` can hold (libhdf5's
/// `cum_max_nrec`). See [`node_info`].
fn cum_max_records(
node_size: u32,
record_size: u16,
offset_size: u8,
max_leaf_nrec: u64,
depth: u16,
) -> u64 {
node_info_from_leaf(node_size, record_size, offset_size, max_leaf_nrec, depth)
.last()
.map_or(max_leaf_nrec, |n| n.cum_max_nrec)
}
/// Capacity of a B-tree v2 node at one depth, as libhdf5 computes it
/// (`H5B2__hdr_init`'s `node_info`).
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub(crate) struct NodeInfo {
/// Most records one node at this depth holds.
pub(crate) max_nrec: u64,
/// Most records a subtree rooted at this depth holds.
pub(crate) cum_max_nrec: u64,
/// Bytes a subtree's total record count takes in a pointer to a node
/// at this depth (0 for a leaf, whose count is its own).
pub(crate) cum_max_nrec_size: usize,
}
/// Node capacities for depths `0..=depth` (entry `d` for depth `d`): a leaf
/// holds `max_nrec(0)` records; an internal node at depth `d` holds
/// `max_nrec(d)` records and `max_nrec(d) + 1` subtrees of depth `d - 1`,
/// where `max_nrec(d)` is what fits in a node once each record is paired
/// with a child pointer of the width depth `d` needs (address, the child's
/// record count in the width a *leaf's* maximum needs, and below the first
/// internal level the child subtree's total in the width its maximum
/// needs), with one pointer more than records.
pub(crate) fn node_info(
node_size: u32,
record_size: u16,
offset_size: u8,
depth: u16,
) -> Vec<NodeInfo> {
let max_leaf = max_records_leaf(node_size, record_size);
node_info_from_leaf(node_size, record_size, offset_size, max_leaf, depth)
}
fn node_info_from_leaf(
node_size: u32,
record_size: u16,
offset_size: u8,
max_leaf_nrec: u64,
depth: u16,
) -> Vec<NodeInfo> {
// Internal node overhead: signature(4) + version(1) + type(1) + checksum(4).
const PREFIX: u64 = 10;
let nrec_width = bytes_for_max_records(max_leaf_nrec) as u64;
let mut info = Vec::with_capacity(usize::from(depth) + 1);
info.push(NodeInfo {
max_nrec: max_leaf_nrec,
cum_max_nrec: max_leaf_nrec,
cum_max_nrec_size: 0,
});
for d in 1..=depth {
let below = info[usize::from(d) - 1];
let ptr = u64::from(offset_size)
+ nrec_width
+ if d > 1 {
below.cum_max_nrec_size as u64
} else {
0
};
let max_nrec = u64::from(node_size)
.saturating_sub(PREFIX)
.saturating_sub(ptr)
/ (u64::from(record_size) + ptr).max(1);
let cum = max_nrec
.saturating_add(1)
.saturating_mul(below.cum_max_nrec)
.saturating_add(max_nrec);
info.push(NodeInfo {
max_nrec,
cum_max_nrec: cum,
cum_max_nrec_size: bytes_for_max_records(cum),
});
}
info
}
#[cfg(test)]
@@ -466,6 +813,132 @@ mod tests {
buf
}
/// An internal node laid out exactly as `collect_internal_records` will
/// read it at `depth`: `records` zeroed records, then `children` pointers,
/// all to `child_addr` claiming `child_nrec` records.
fn internal_node(
depth: u16,
node_size: u32,
record_size: u16,
records: usize,
children: usize,
child_addr: u64,
child_nrec: u64,
) -> Vec<u8> {
let max_leaf = max_records_leaf(node_size, record_size);
let nrec_width = bytes_for_max_records(max_leaf);
let total_width = if depth > 1 {
bytes_for_max_records(cum_max_records(
node_size,
record_size,
8,
max_leaf,
depth - 1,
))
} else {
0
};
let mut buf = b"BTIN".to_vec();
buf.extend_from_slice(&[0, 5]);
buf.resize(buf.len() + records * record_size as usize, 0);
for _ in 0..children {
buf.extend_from_slice(&child_addr.to_le_bytes());
buf.extend_from_slice(&child_nrec.to_le_bytes()[..nrec_width]);
buf.resize(buf.len() + total_width, 0);
}
let sum = crate::checksum::jenkins_lookup3(&buf);
buf.extend_from_slice(&sum.to_le_bytes());
buf
}
fn header(depth: u16, root: u64, root_nrec: u16, total: u64) -> BTreeV2Header {
BTreeV2Header {
tree_type: 5,
node_size: 512,
record_size: 8,
depth,
root_node_address: root,
num_records_in_root: root_nrec,
total_records: total,
}
}
#[test]
fn a_node_that_is_its_own_child_is_rejected_not_recursed() {
// One internal node whose two children are itself, under a header
// claiming the deepest tree a u16 allows. The layout stops depending
// on depth once the subtree-total width saturates, so every level
// parses cleanly and recursion runs ~65 000 frames deep: before the
// cap this overflowed the stack and aborted the process, from a file
// of under 100 bytes.
let mut data = internal_node(u16::MAX, 512, 8, 1, 2, 0, 1);
data.resize(4096, 0);
let result = collect_btree_v2_records(&data, &header(u16::MAX, 0, 1, 1), 8, 8);
assert!(result.is_err(), "{result:?}");
}
#[test]
fn a_shared_subtree_cannot_multiply_the_work() {
// A chain of distinct levels, each node's children all pointing at the
// single node below, ending in a real leaf. Every node parses and
// nothing is cyclic, yet the leaf is reached fan-out^depth times: 62
// children over 4 levels is ~15 million leaf visits from a few
// kilobytes. A valid tree cannot hold more records than the file has
// room for, so that bounds the traversal instead.
let (node_size, record_size) = (512u32, 8u16);
let fanout = 62usize;
let depth = 4u16;
let leaf = build_leaf_node(5, &[&[0u8; 8][..]]);
// Lay out root first, then each lower level, then the leaf.
let mut nodes: Vec<Vec<u8>> = Vec::new();
let mut addrs = Vec::new();
let mut at = 0u64;
let mut sizes = Vec::new();
for d in (1..=depth).rev() {
let n = internal_node(d, node_size, record_size, fanout - 1, fanout, 0, 0);
sizes.push(n.len());
}
for size in &sizes {
addrs.push(at);
at += *size as u64;
}
let leaf_addr = at;
for (i, d) in (1..=depth).rev().enumerate() {
let (child, child_nrec) = if d == 1 {
(leaf_addr, 1)
} else {
(addrs[i + 1], fanout as u64 - 1)
};
nodes.push(internal_node(
d,
node_size,
record_size,
fanout - 1,
fanout,
child,
child_nrec,
));
}
let mut data: Vec<u8> = nodes.concat();
data.extend_from_slice(&leaf);
data.resize(data.len() + 64, 0);
let started = std::time::Instant::now();
let result =
collect_btree_v2_records(&data, &header(depth, 0, fanout as u16 - 1, u64::MAX), 8, 8);
assert!(
result.is_err(),
"expected a refusal, got {} records",
result.map_or(0, |r| r.len())
);
assert!(
started.elapsed() < std::time::Duration::from_secs(2),
"took {:?}",
started.elapsed()
);
}
#[test]
fn parse_header() {
let data = build_btree_v2_header(5, 512, 11, 0, 0x1000, 3, 3, 8, 8);
@@ -522,4 +995,18 @@ mod tests {
let records = collect_btree_v2_records(&header, &hdr, 8, 8).unwrap();
assert!(records.is_empty());
}
#[test]
fn subtree_capacity_matches_libhdf5() {
// A link-name index (11-byte records, 512-byte nodes, 8-byte
// addresses): libhdf5's H5B2__hdr_init gives 45 records per leaf,
// then cum_max_nrec 1 149 at depth 1 and 26 449 at depth 2 — two
// bytes of subtree count in a depth-3 root's child pointers, where
// leaf_max^3 = 91 125 would need three.
let leaf = max_records_leaf(512, 11);
assert_eq!(leaf, 45);
assert_eq!(cum_max_records(512, 11, 8, leaf, 0), 45);
assert_eq!(cum_max_records(512, 11, 8, leaf, 1), 1_149);
assert_eq!(cum_max_records(512, 11, 8, leaf, 2), 26_449);
}
}
crates/clawhdf5-format/src/btree_v2_write.rs
+503
View File
@@ -0,0 +1,503 @@
//! Writing version-2 B-trees: a header (`BTHD`) and its nodes, leaves
//! (`BTLF`) and, for more records than one leaf holds, internal nodes
//! (`BTIN`) to any depth.
//!
//! Node capacities come from [`crate::btree_v2::node_info`], the arithmetic
//! libhdf5 uses (`H5B2__hdr_init`) and the reader decodes pointers with, so
//! the pointer widths the writer encodes are the ones every reader expects.
use crate::addr::saturating_usize;
#[cfg(not(feature = "std"))]
use alloc::{format, vec, vec::Vec};
use crate::btree_v2::{NodeInfo, bytes_for_max_records, node_info};
use crate::checksum::jenkins_lookup3;
use crate::error::FormatError;
/// How a B-tree is laid out: its record type and node geometry, as the
/// header records them.
#[derive(Debug, Clone, Copy)]
pub(crate) struct BTreeV2Params {
/// Record type (5: link names, 6: link creation order, 8: attribute
/// names, 9: attribute creation order, 10/11: chunks).
pub(crate) tree_type: u8,
/// Bytes per node.
pub(crate) node_size: u32,
/// Bytes per record.
pub(crate) record_size: u16,
/// Split and merge percentages. The writer fills nodes itself; these
/// only tell libhdf5 when to split and merge as it modifies the tree.
pub(crate) split_percent: u8,
pub(crate) merge_percent: u8,
}
/// Size of a B-tree v2 header.
pub(crate) fn header_size(offset_size: u8, length_size: u8) -> usize {
4 + 1 + 1 + 4 + 2 + 2 + 1 + 1 + offset_size as usize + 2 + length_size as usize + 4
}
/// Deepest tree the writer builds. Even at the smallest fan-out libhdf5's
/// arithmetic allows, a few levels hold more records than any file could.
const MAX_WRITE_DEPTH: u16 = 32;
/// Write a B-tree v2 holding `records` (`record_size` bytes each,
/// concatenated, already in the tree's key order) at `addr`: the header,
/// then its nodes, each `node_size` bytes. No records gives a header with
/// an undefined root.
///
/// The tree is as shallow as the node size allows: a single leaf when the
/// records fit one, otherwise internal nodes above leaves. Records are
/// spread evenly over each node's children, so every node but the root is
/// at least about half full (above libhdf5's merge threshold, which is below
/// half), and each node holds at most its depth's maximum.
pub(crate) fn build_btree_v2(
p: BTreeV2Params,
records: &[u8],
addr: u64,
offset_size: u8,
length_size: u8,
) -> Result<Vec<u8>, FormatError> {
let rs = usize::from(p.record_size);
if rs == 0 || !records.len().is_multiple_of(rs) {
return Err(FormatError::SerializationError(format!(
"B-tree v2 records are {} bytes, not a multiple of the record size {rs}",
records.len()
)));
}
let n = (records.len() / rs) as u64;
let hdr_len = header_size(offset_size, length_size);
// The shallowest depth whose subtree can hold every record.
let mut info = node_info(p.node_size, p.record_size, offset_size, 0);
let max_leaf = info[0].max_nrec;
if max_leaf == 0 || max_leaf > u64::from(u16::MAX) {
return Err(FormatError::SerializationError(format!(
"a {}-byte B-tree v2 node holds {max_leaf} {}-byte records; \
a node holds 1 to 65535",
p.node_size, p.record_size
)));
}
let mut depth = 0u16;
while info[usize::from(depth)].cum_max_nrec < n {
depth += 1;
if depth > MAX_WRITE_DEPTH {
return Err(FormatError::SerializationError(format!(
"{n} records do not fit a B-tree v2 of {}-byte nodes",
p.node_size
)));
}
info = node_info(p.node_size, p.record_size, offset_size, depth);
let max = info[usize::from(depth)].max_nrec;
if max == 0 || max > u64::from(u16::MAX) {
return Err(FormatError::SerializationError(format!(
"a {}-byte B-tree v2 internal node holds {max} records; \
a node holds 1 to 65535",
p.node_size
)));
}
}
let mut w = TreeWriter {
p,
records,
info: &info,
nrec_width: bytes_for_max_records(max_leaf),
offset_size,
first_node: addr + hdr_len as u64,
nodes: Vec::new(),
};
let root = (n > 0)
.then(|| w.node(depth, 0, saturating_usize(n)))
.transpose()?;
let mut out = Vec::with_capacity(hdr_len + w.nodes.len() * p.node_size as usize);
out.extend_from_slice(b"BTHD");
out.push(0); // version
out.push(p.tree_type);
out.extend_from_slice(&p.node_size.to_le_bytes());
out.extend_from_slice(&p.record_size.to_le_bytes());
out.extend_from_slice(&depth.to_le_bytes());
out.push(p.split_percent);
out.push(p.merge_percent);
match root {
Some(r) => push_uint(&mut out, r.addr, offset_size as usize),
None => out.extend(core::iter::repeat_n(0xFF, offset_size as usize)),
}
let root_nrec = root.map_or(0, |r| r.nrec);
out.extend_from_slice(&(root_nrec as u16).to_le_bytes());
push_uint(&mut out, n, length_size as usize);
let sum = jenkins_lookup3(&out);
out.extend_from_slice(&sum.to_le_bytes());
debug_assert_eq!(out.len(), hdr_len);
for node in &w.nodes {
out.extend_from_slice(node);
}
Ok(out)
}
/// A written node, as its parent points at it.
#[derive(Debug, Clone, Copy)]
struct NodeRef {
addr: u64,
/// Records in the node itself.
nrec: u64,
/// Records in the subtree it roots.
all_nrec: u64,
}
struct TreeWriter<'a> {
p: BTreeV2Params,
records: &'a [u8],
info: &'a [NodeInfo],
/// Width of a child's record count: what a leaf's maximum needs.
nrec_width: usize,
offset_size: u8,
/// Address of the first node (right after the header).
first_node: u64,
/// Nodes in file order (children before their parent).
nodes: Vec<Vec<u8>>,
}
impl TreeWriter<'_> {
fn record(&self, i: usize) -> &[u8] {
let rs = usize::from(self.p.record_size);
&self.records[i * rs..(i + 1) * rs]
}
fn push_node(&mut self, mut node: Vec<u8>) -> u64 {
// The checksum covers the node up to it, not the padding after.
let sum = jenkins_lookup3(&node);
node.extend_from_slice(&sum.to_le_bytes());
debug_assert!(node.len() <= self.p.node_size as usize);
node.resize(self.p.node_size as usize, 0);
let addr = self.first_node + self.nodes.len() as u64 * u64::from(self.p.node_size);
self.nodes.push(node);
addr
}
/// Write the subtree of `depth` holding records `first..first + n`.
fn node(&mut self, depth: u16, first: usize, n: usize) -> Result<NodeRef, FormatError> {
let rs = usize::from(self.p.record_size);
let mut node = Vec::with_capacity(self.p.node_size as usize);
if depth == 0 {
debug_assert!(n as u64 <= self.info[0].max_nrec);
node.extend_from_slice(b"BTLF");
node.push(0); // version
node.push(self.p.tree_type);
node.extend_from_slice(&self.records[first * rs..(first + n) * rs]);
let addr = self.push_node(node);
return Ok(NodeRef {
addr,
nrec: n as u64,
all_nrec: n as u64,
});
}
// As few children as hold the records, at least two, with the
// records spread evenly: `k` children and `k - 1` records between
// them.
let below = self.info[usize::from(depth) - 1].cum_max_nrec;
let k = (n as u64 + 1).div_ceil(below + 1).max(2);
let max = self.info[usize::from(depth)].max_nrec;
if k - 1 > max || (n as u64) < k - 1 + k {
return Err(FormatError::SerializationError(format!(
"cannot spread {n} B-tree v2 records over {k} children at depth {depth}"
)));
}
let k = saturating_usize(k);
let in_children = n - (k - 1);
let (base, extra) = (in_children / k, in_children % k);
let mut children = Vec::with_capacity(k);
let mut separators = Vec::with_capacity(k - 1);
let mut next = first;
for c in 0..k {
let m = base + usize::from(c < extra);
children.push(self.node(depth - 1, next, m)?);
next += m;
if c + 1 < k {
separators.push(next);
next += 1;
}
}
debug_assert_eq!(next, first + n);
node.extend_from_slice(b"BTIN");
node.push(0); // version
node.push(self.p.tree_type);
for &s in &separators {
node.extend_from_slice(self.record(s));
}
let total_width = if depth > 1 {
self.info[usize::from(depth) - 1].cum_max_nrec_size
} else {
0
};
for c in &children {
push_uint(&mut node, c.addr, self.offset_size as usize);
push_uint(&mut node, c.nrec, self.nrec_width);
if depth > 1 {
push_uint(&mut node, c.all_nrec, total_width);
}
}
let addr = self.push_node(node);
Ok(NodeRef {
addr,
nrec: (k - 1) as u64,
all_nrec: n as u64,
})
}
}
/// Append `v` as a `width`-byte little-endian integer.
fn push_uint(buf: &mut Vec<u8>, v: u64, width: usize) {
let bytes = v.to_le_bytes();
buf.extend_from_slice(&bytes[..width.min(8)]);
buf.extend(vec![0u8; width.saturating_sub(8)]);
}
#[cfg(test)]
mod tests {
use super::*;
use crate::btree_v2::{BTreeV2Header, collect_btree_v2_records};
fn params(node_size: u32, record_size: u16) -> BTreeV2Params {
BTreeV2Params {
tree_type: 5,
node_size,
record_size,
split_percent: 100,
merge_percent: 40,
}
}
/// `n` 11-byte records: a big-endian counter, so byte order is key order.
fn records(n: usize, rs: usize) -> Vec<u8> {
let mut out = Vec::with_capacity(n * rs);
for i in 0..n {
let mut r = vec![0u8; rs];
r[..8].copy_from_slice(&(i as u64).to_be_bytes());
out.extend_from_slice(&r);
}
out
}
fn roundtrip(node_size: u32, rs: u16, n: usize, os: u8, ls: u8) -> BTreeV2Header {
let recs = records(n, usize::from(rs));
let base = 4096u64;
let tree = build_btree_v2(params(node_size, rs), &recs, base, os, ls).unwrap();
let mut file = vec![0u8; base as usize];
file.extend_from_slice(&tree);
let hdr = BTreeV2Header::parse(&file, base as usize, os, ls).unwrap();
assert_eq!(hdr.total_records, n as u64);
let got = collect_btree_v2_records(&file, &hdr, os, ls).unwrap();
assert_eq!(got.len(), n);
let flat: Vec<u8> = got.into_iter().flat_map(|r| r.data).collect();
assert_eq!(flat, recs, "node {node_size} rs {rs} n {n}");
hdr
}
#[test]
fn one_leaf_then_deeper_trees_read_back_in_order() {
// 512-byte nodes of 11-byte records: 45 per leaf, 1149 at depth 1,
// 26 449 at depth 2.
let info = node_info(512, 11, 8, 3);
assert_eq!(
info.iter().map(|i| i.cum_max_nrec).collect::<Vec<_>>(),
[45, 1149, 26_449, 608_349]
);
for (n, depth) in [
(0, 0),
(1, 0),
(45, 0),
(46, 1),
(1149, 1),
(1150, 2),
(26_449, 2),
(26_450, 3),
(100_000, 3),
] {
let hdr = roundtrip(512, 11, n, 8, 8);
assert_eq!(hdr.depth, depth, "{n} records");
}
}
#[test]
fn pointer_widths_follow_the_offset_and_length_sizes() {
for (os, ls) in [(4, 4), (8, 4), (4, 8), (2, 2)] {
roundtrip(512, 11, 5000, os, ls);
}
// Wide counts: a leaf of 2048 bytes / 9-byte records (226, one byte)
// and deeper subtree totals of three bytes.
roundtrip(2048, 9, 300_000, 8, 8);
}
#[test]
fn every_node_is_within_its_capacity_and_above_the_merge_threshold() {
let rs = 17u16;
let n = 70_000usize;
let info = node_info(512, rs, 8, 3);
let recs = records(n, usize::from(rs));
let tree = build_btree_v2(params(512, rs), &recs, 0, 8, 8).unwrap();
let hdr_len = header_size(8, 8);
let nodes = (tree.len() - hdr_len) / 512;
for i in 0..nodes {
let node = &tree[hdr_len + i * 512..hdr_len + (i + 1) * 512];
let sig = &node[..4];
if sig == b"BTLF" {
continue; // counts checked through the parents below
}
assert_eq!(sig, b"BTIN");
}
// Walk from the header: each child's count within [40%, 100%].
let hdr = BTreeV2Header::parse(&tree, 0, 8, 8).unwrap();
assert_eq!(hdr.depth, 3);
assert!(u64::from(hdr.num_records_in_root) <= info[3].max_nrec);
fn walk(tree: &[u8], addr: usize, nrec: usize, depth: usize, info: &[NodeInfo], rs: usize) {
if depth == 0 {
return;
}
let nrec_w = bytes_for_max_records(info[0].max_nrec);
let tot_w = if depth > 1 {
info[depth - 1].cum_max_nrec_size
} else {
0
};
let mut pos = addr + 6 + nrec * rs;
for _ in 0..=nrec {
let a = u64::from_le_bytes(tree[pos..pos + 8].try_into().unwrap()) as usize;
pos += 8;
let mut c = 0usize;
for b in 0..nrec_w {
c |= usize::from(tree[pos + b]) << (8 * b);
}
pos += nrec_w + tot_w;
let max = info[depth - 1].max_nrec as usize;
assert!(c <= max && c * 100 > max * 40, "{c} of {max}");
walk(tree, a, c, depth - 1, info, rs);
}
}
walk(
&tree,
hdr.root_node_address as usize,
usize::from(hdr.num_records_in_root),
3,
&info,
usize::from(rs),
);
assert!(nodes > 0);
}
/// Descending to a key range finds exactly the records a full read
/// holds in it — runs of equal keys that straddle node boundaries
/// included — at every depth, and nothing for keys not in the tree.
#[test]
fn a_key_range_search_matches_a_full_scan() {
use crate::btree_v2::find_btree_v2_records;
use core::cmp::Ordering;
let rs = 11usize;
// Keys 0, 0, 0, 2, 2, 2, 4, ...: runs of three, odd keys missing.
for n in [1usize, 45, 46, 1150, 30_000] {
let mut recs = Vec::with_capacity(n * rs);
for i in 0..n {
let mut r = vec![0u8; rs];
r[..8].copy_from_slice(&((i / 3 * 2) as u64).to_be_bytes());
r[8..].copy_from_slice(&[(i % 3) as u8, 0, 0]);
recs.extend_from_slice(&r);
}
let base = 4096u64;
let tree = build_btree_v2(params(512, 11), &recs, base, 8, 8).unwrap();
let mut file = vec![0u8; base as usize];
file.extend_from_slice(&tree);
let hdr = BTreeV2Header::parse(&file, base as usize, 8, 8).unwrap();
let all = collect_btree_v2_records(&file, &hdr, 8, 8).unwrap();
let key = |r: &[u8]| u64::from_be_bytes(r[..8].try_into().unwrap());
let last = key(&all[n - 1].data);
let probes = (0..=last + 1).step_by(if n > 1000 { 37 } else { 1 });
for k in probes.chain([last, last + 1, u64::MAX]) {
let found =
find_btree_v2_records(&file, &hdr, 8, &mut |r: &[u8]| key(r).cmp(&k)).unwrap();
let want: Vec<&[u8]> = all
.iter()
.map(|r| r.data.as_slice())
.filter(|r| key(r) == k)
.collect();
let got: Vec<&[u8]> = found.iter().map(|r| r.data.as_slice()).collect();
assert_eq!(got, want, "n {n} key {k}");
assert_eq!(
got.len(),
if k % 2 == 0 && k <= last {
want.len()
} else {
0
}
);
}
// Every record, or none, when the whole tree is in or out of range.
let every = find_btree_v2_records(&file, &hdr, 8, &mut |_| Ordering::Equal).unwrap();
assert_eq!(every.len(), n);
let none = find_btree_v2_records(&file, &hdr, 8, &mut |_| Ordering::Less).unwrap();
assert!(none.is_empty());
}
}
/// A two-level tree read through a `read_at`-only storage gives what
/// the slice gives — records, descents and errors — whole, truncated
/// at every length, and with each byte of its nodes flipped, and each
/// node costs one read.
#[test]
fn storage_reads_match_slice_reads() {
use crate::btree_v2::{
collect_btree_v2_records_in, find_btree_v2_records, find_btree_v2_records_in,
};
use crate::storage::CountingStorage;
let (rs, n, base) = (11usize, 120usize, 64usize);
let recs = records(n, rs);
let tree = build_btree_v2(params(128, 11), &recs, base as u64, 8, 8).unwrap();
let mut whole = vec![0u8; base];
whole.extend_from_slice(&tree);
let hdr = BTreeV2Header::parse(&whole, base, 8, 8).unwrap();
assert!(hdr.depth >= 1, "{hdr:?}");
let key = |r: &[u8]| u64::from_be_bytes(r[..8].try_into().unwrap());
let mut files = Vec::new();
for cut in base..=whole.len() {
files.push(whole[..cut].to_vec());
}
for at in base..whole.len() {
let mut bad = whole.clone();
bad[at] ^= 0x5a;
files.push(bad);
}
let mut ok = 0;
for f in &files {
let st = CountingStorage::new(f.clone());
let want_h = BTreeV2Header::parse(f, base, 8, 8);
let got_h = BTreeV2Header::parse_in(&st, base as u64, 8, 8);
assert_eq!(format!("{got_h:?}"), format!("{want_h:?}"));
// The nodes of the intact header, over each damaged file.
let want = collect_btree_v2_records(f, &hdr, 8, 8);
st.reset();
let got = collect_btree_v2_records_in(&st, &hdr, 8, 8);
assert_eq!(format!("{got:?}"), format!("{want:?}"));
if want.is_ok() {
ok += 1;
assert!(st.reads() <= 1 + n as u64 / 3, "{} reads", st.reads());
}
for k in [0u64, 7, 60, 119, 500] {
let want = find_btree_v2_records(f, &hdr, 8, &mut |r: &[u8]| key(r).cmp(&k));
let got = find_btree_v2_records_in(&st, &hdr, 8, &mut |r: &[u8]| key(r).cmp(&k));
assert_eq!(format!("{got:?}"), format!("{want:?}"));
}
}
assert!(ok > 1);
}
#[test]
fn a_node_too_small_or_too_big_is_an_error() {
assert!(build_btree_v2(params(16, 11), &records(1, 11), 0, 8, 8).is_err());
// A leaf with room for more than 65 535 records.
assert!(build_btree_v2(params(1 << 20, 11), &records(1, 11), 0, 8, 8).is_err());
// Records that are not whole.
assert!(build_btree_v2(params(512, 11), &[0u8; 12], 0, 8, 8).is_err());
}
}
crates/clawhdf5-format/src/bulk_alloc.rs
+79
View File
@@ -0,0 +1,79 @@
//! Large output buffers backed by transparent huge pages where the OS offers
//! them.
//!
//! A fresh multi-megabyte `Vec` is mapped lazily by the kernel: the first
//! write to each 4 KiB page takes a page fault, and the kernel zeroes the page
//! before handing it over. For a 64 MiB read that is 16384 faults, and they
//! cost far more than the copy that fills the buffer — single-threaded
//! contiguous reads ran at about a quarter of h5py's speed because of them.
//! numpy (so h5py) avoids this by asking for transparent huge pages
//! (`madvise(MADV_HUGEPAGE)`) on every allocation of 4 MiB or more, which
//! turns 512 faults into one; this module does the same.
//!
//! The advice only changes how the pages are backed, never their contents, so
//! it is harmless when it cannot be honoured (THP disabled, not Linux, a
//! region that is part of the heap): the buffer is then exactly what it would
//! have been without it.
#[cfg(not(feature = "std"))]
use alloc::vec::Vec;
/// Buffers smaller than this are left alone (numpy uses the same threshold).
#[cfg(any(target_os = "linux", test))]
pub(crate) const HUGE_PAGE_THRESHOLD: usize = 4 << 20;
/// Advise the kernel to back `[ptr, ptr + len)` with transparent huge pages,
/// when `len` is large enough to benefit. Call it before the first write so
/// the faults happen at huge-page granularity.
#[inline]
pub(crate) fn advise_huge_pages(ptr: *const u8, len: usize) {
#[cfg(target_os = "linux")]
if len >= HUGE_PAGE_THRESHOLD {
const PAGE: usize = 4096;
let start = (ptr as usize).next_multiple_of(PAGE);
let end = (ptr as usize + len) & !(PAGE - 1);
if end > start {
// SAFETY: `[start, end)` lies inside an allocation of `len` bytes
// at `ptr` that the caller owns, and is page aligned as madvise
// requires. MADV_HUGEPAGE does not change the memory's contents or
// validity; on failure (EINVAL when THP is compiled out, etc.) the
// region is simply left as it was, so the result is ignored.
unsafe {
libc::madvise(start as *mut libc::c_void, end - start, libc::MADV_HUGEPAGE);
}
}
}
#[cfg(not(target_os = "linux"))]
let _ = (ptr, len);
}
/// `Vec::with_capacity(count)` for a buffer about to be filled in bulk, with
/// huge-page advice when it is large (see the module docs).
#[inline]
pub(crate) fn vec_for_bulk<T>(count: usize) -> Vec<T> {
let v: Vec<T> = Vec::with_capacity(count);
advise_huge_pages(
v.as_ptr().cast::<u8>(),
v.capacity().saturating_mul(core::mem::size_of::<T>()),
);
v
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn bulk_vec_is_an_ordinary_vec() {
for count in [0usize, 1, 1000, HUGE_PAGE_THRESHOLD / 4 + 3] {
let mut v: Vec<u32> = vec_for_bulk(count);
assert!(v.capacity() >= count);
v.extend((0..count as u32).map(|i| i.wrapping_mul(2654435761)));
assert!(
v.iter()
.enumerate()
.all(|(i, &x)| x == (i as u32).wrapping_mul(2654435761))
);
}
}
}
crates/clawhdf5-format/src/checksum.rs
+52
View File
@@ -14,6 +14,45 @@ pub fn jenkins_lookup3(data: &[u8]) -> u32 {
hashlittle(data, 0)
}
/// HDF5's Fletcher-32 checksum, as the Fletcher-32 I/O filter (filter id 3)
/// stores it after each chunk.
///
/// A line-for-line port of `H5_checksum_fletcher32` (H5checksum.c, libhdf5
/// 1.8 through 1.14): big-endian 16-bit words summed in blocks of 360, each
/// sum reduced after a block by the ones'-complement fold
/// `(s & 0xffff) + (s >> 16)` rather than `% 65535`, an odd trailing byte
/// taken as the high byte of a last word, and a final fold of both sums.
/// The fold and `% 65535` differ whenever a sum is a non-zero multiple of
/// 65535: the fold leaves 0xffff where the modulo gives 0, so the two
/// disagree on about one chunk in 32768 and libhdf5 rejects the other's
/// checksum. This must stay the only implementation.
pub fn fletcher32(data: &[u8]) -> u32 {
let mut sum1: u32 = 0;
let mut sum2: u32 = 0;
// 360 words keep both sums inside 32 bits between folds (the bound
// libhdf5 uses: after a fold sum1 < 0x10200, so sum2 stays below
// 360 * 361 / 2 * 0xffff + 360 * 0x10200 + 0x1fffe < 2^32). The adds wrap
// like the C unsigned arithmetic all the same.
let (words, odd) = data.as_chunks::<2>();
for block in words.chunks(360) {
for w in block {
sum1 = sum1.wrapping_add((u32::from(w[0]) << 8) | u32::from(w[1]));
sum2 = sum2.wrapping_add(sum1);
}
sum1 = (sum1 & 0xffff) + (sum1 >> 16);
sum2 = (sum2 & 0xffff) + (sum2 >> 16);
}
if let [last] = odd {
sum1 = sum1.wrapping_add(u32::from(*last) << 8);
sum2 = sum2.wrapping_add(sum1);
sum1 = (sum1 & 0xffff) + (sum1 >> 16);
sum2 = (sum2 & 0xffff) + (sum2 >> 16);
}
sum1 = (sum1 & 0xffff) + (sum1 >> 16);
sum2 = (sum2 & 0xffff) + (sum2 >> 16);
(sum2 << 16) | sum1
}
/// Compute CRC32 (IEEE / ISO 3309) over data.
///
/// When the `fast-checksum` feature is enabled, this uses hardware CRC32
@@ -207,6 +246,19 @@ fn hashlittle(data: &[u8], initval: u32) -> u32 {
mod tests {
use super::*;
/// Values of libhdf5's `H5_checksum_fletcher32` (h5py 3.x's bundled
/// libhdf5, called through ctypes). The first three are sums that are
/// multiples of 65535, where `% 65535` gave 0 instead of 0xffff.
#[test]
fn fletcher32_matches_libhdf5() {
assert_eq!(fletcher32(&[0x00, 0x01, 0xff, 0xfe]), 0x0001_ffff);
assert_eq!(fletcher32(&[0xff; 720]), 0xffff_ffff);
assert_eq!(fletcher32(&[0xff; 721]), 0xff00_ff00);
assert_eq!(fletcher32(&[0xff; 1441]), 0xff00_ff00);
assert_eq!(fletcher32(&[]), 0);
assert_eq!(fletcher32(&[7]), 0x0700_0700);
}
#[test]
fn empty_input() {
// Empty input should return the initial state after no mixing
crates/clawhdf5-format/src/chunk_cache.rs
+520 -270
View File
@@ -223,13 +223,32 @@ pub const DEFAULT_CACHE_BYTES: usize = 16 * 1024 * 1024; // 16 MiB
/// coordinate map and reduces collision chains compared to power-of-two sizes.
pub const DEFAULT_MAX_SLOTS: usize = 521;
/// Most datasets whose chunk index a [`ChunkCache`] keeps at once.
pub const MAX_INDEXED_DATASETS: usize = 64;
/// Most chunk-index entries, summed over all datasets, a [`ChunkCache`] keeps.
/// Least-recently-used datasets' indexes are dropped past this (the dataset
/// being read is always kept), so a file with many or huge chunked datasets
/// cannot grow the cache without bound.
pub const MAX_INDEXED_CHUNKS: usize = 1 << 20;
/// The dataset key the address-less (legacy) methods use when
/// [`ChunkCache::ensure_dataset`] has not been called.
#[cfg(feature = "std")]
const UNBOUND_DATASET: u64 = u64::MAX;
// ---------------------------------------------------------------------------
// LRU entry
// ---------------------------------------------------------------------------
/// Decompressed chunks are keyed by dataset *and* coordinate: every chunked
/// dataset has a chunk at (0, 0, ...), so the coordinate alone is ambiguous.
#[cfg(feature = "std")]
type SlotKey = (u64, ChunkCoord);
#[cfg(feature = "std")]
struct CachedChunk {
coord: ChunkCoord,
key: SlotKey,
/// Shared so a cache hit is a refcount bump, not a copy of the whole
/// (potentially large) decompressed chunk.
data: Arc<CacheAlignedBuffer>,
@@ -237,21 +256,48 @@ struct CachedChunk {
last_access: u64,
}
/// Per-dataset index state.
#[cfg(feature = "std")]
#[derive(Default)]
struct DatasetEntry {
/// Chunk coordinate -> ChunkInfo (offset + size in file).
index: Option<Arc<HashMap<ChunkCoord, ChunkInfo>>>,
/// Pre-built chunk index for O(1) coordinate lookups.
chunk_index: Option<Arc<ChunkIndex>>,
/// Pre-computed chunk layout for fast assembly.
chunk_layout: Option<Arc<ChunkLayout>>,
/// Tick of the last use, for dropping the least recently used dataset.
last_used: u64,
}
#[cfg(feature = "std")]
impl DatasetEntry {
fn weight(&self) -> usize {
self.index.as_ref().map_or(0, |m| m.len())
+ self.chunk_index.as_ref().map_or(0, |c| c.num_chunks())
}
}
// ---------------------------------------------------------------------------
// ChunkCache
// ---------------------------------------------------------------------------
/// A per-dataset chunk cache with hash-based index and LRU eviction.
/// A per-file chunk cache: chunk indexes per dataset, plus an LRU of
/// decompressed chunks, all keyed by dataset.
///
/// # Usage
/// A dataset is identified by the address of its chunk index (B-tree, fixed
/// or extensible array, ...), which is unique within a file. Every method
/// that takes an `addr` works on that dataset only, so threads reading
/// different datasets through one shared cache never see each other's
/// chunks. The address-less methods (`has_index`, `populate_index`,
/// `get_decompressed`, ...) act on the dataset last bound with
/// [`Self::ensure_dataset`]; that binding is shared state, so concurrent
/// readers must use the `*_in` / `*_for` methods instead (the chunked
/// readers in [`crate::chunked_read`] do).
///
/// ```ignore
/// let cache = ChunkCache::new();
/// // Pass &cache to read_chunked_data — it will populate the index lazily.
/// ```
///
/// The cache is wrapped in `Mutex` internally so it can be mutated through
/// shared references (thread-safe).
/// Memory is bounded: decompressed data by `max_bytes`/`max_slots` across
/// all datasets, indexes by [`MAX_INDEXED_DATASETS`] and
/// [`MAX_INDEXED_CHUNKS`].
///
/// Only available with the `std` feature because it requires `std::sync::Mutex`.
#[cfg(feature = "std")]
@@ -261,26 +307,20 @@ pub struct ChunkCache {
#[cfg(feature = "std")]
struct CacheInner {
/// Hash index: chunk coordinate -> ChunkInfo (offset + size in file).
/// Populated once per dataset on first access.
index: Option<HashMap<ChunkCoord, ChunkInfo>>,
/// Per-dataset chunk indexes, keyed by chunk-index address.
datasets: HashMap<u64, DatasetEntry>,
/// Address of the dataset (its chunk-index base address) that the cached
/// index, chunk index, layout, and decompressed slots currently belong to.
/// The cache is shared per file across datasets, so every cached-read entry
/// checks this and resets the per-dataset state when the dataset changes —
/// otherwise one dataset's chunk index (with its own rank) would be reused
/// for another, corrupting reads.
index_addr: Option<u64>,
/// Dataset the address-less methods act on (see `ensure_dataset`).
current: Option<u64>,
/// LRU cache of decompressed chunk data.
slots: Vec<CachedChunk>,
/// Coordinate -> index into `slots`, for O(1) lookup instead of a linear
/// Key -> index into `slots`, for O(1) lookup instead of a linear
/// scan. Kept in sync with `slots` on every insert/evict/clear — in
/// particular, `slots.swap_remove(i)` moves the last element into slot
/// `i`, so the moved element's index entry must be updated too.
slot_index: HashMap<ChunkCoord, usize>,
slot_index: HashMap<SlotKey, usize>,
/// Current total bytes of cached decompressed data.
current_bytes: usize,
@@ -294,17 +334,145 @@ struct CacheInner {
/// Monotonic counter for LRU ordering.
tick: u64,
/// Last accessed chunk coordinate (for sequential detection).
last_coord: Option<ChunkCoord>,
/// Last accessed chunk (for sequential detection).
last_coord: Option<SlotKey>,
/// Access pattern statistics.
stats: AccessStats,
}
/// Pre-built chunk index for O(1) coordinate lookups.
chunk_index: Option<ChunkIndex>,
#[cfg(feature = "std")]
impl CacheInner {
fn current(&self) -> u64 {
self.current.unwrap_or(UNBOUND_DATASET)
}
/// Pre-computed chunk layout for fast assembly.
chunk_layout: Option<ChunkLayout>,
fn touch(&mut self, addr: u64) -> &mut DatasetEntry {
self.tick += 1;
let tick = self.tick;
let entry = self.datasets.entry(addr).or_default();
entry.last_used = tick;
entry
}
fn entry(&self, addr: u64) -> Option<&DatasetEntry> {
self.datasets.get(&addr)
}
/// Drop least-recently-used datasets' indexes (never `keep`'s) until the
/// dataset and chunk-entry budgets hold.
fn trim_datasets(&mut self, keep: u64) {
loop {
let total: usize = self.datasets.values().map(DatasetEntry::weight).sum();
if self.datasets.len() <= MAX_INDEXED_DATASETS && total <= MAX_INDEXED_CHUNKS {
return;
}
let victim = self
.datasets
.iter()
.filter(|(a, _)| **a != keep)
.min_by_key(|(_, e)| e.last_used)
.map(|(a, _)| *a);
match victim {
Some(a) => {
self.datasets.remove(&a);
}
None => return,
}
}
}
fn get_decompressed(&mut self, addr: u64, coord: &[u64]) -> Option<Arc<CacheAlignedBuffer>> {
self.tick += 1;
let tick = self.tick;
// Track sequential vs random access
let is_sequential = self.last_coord.as_ref().is_some_and(|(prev_addr, prev)| {
// Sequential if exactly one dimension changed
let changes: usize = prev
.iter()
.zip(coord.iter())
.filter(|(a, b)| a != b)
.count();
*prev_addr == addr && changes <= 1
});
if is_sequential {
self.stats.sequential_count += 1;
} else if self.last_coord.is_some() {
self.stats.random_count += 1;
}
let key: SlotKey = (addr, coord.to_vec());
let found = if let Some(&idx) = self.slot_index.get(&key) {
self.slots[idx].last_access = tick;
Some(Arc::clone(&self.slots[idx].data))
} else {
None
};
self.last_coord = Some(key);
if let Some(ref data) = found {
self.stats.hits += 1;
self.stats.bytes_read += data.len() as u64;
} else {
self.stats.misses += 1;
}
found
}
fn put_decompressed(
&mut self,
key: SlotKey,
data: Arc<CacheAlignedBuffer>,
) -> Arc<CacheAlignedBuffer> {
let data_len = data.len();
// Don't cache if single chunk exceeds budget — still return the data
// to the caller, just don't retain it.
if data_len > self.max_bytes {
return data;
}
// Check if already present
self.tick += 1;
let tick = self.tick;
if let Some(&idx) = self.slot_index.get(&key) {
self.slots[idx].last_access = tick;
return Arc::clone(&self.slots[idx].data); // already cached
}
// Evict until we have room
while self.slots.len() >= self.max_slots
|| (self.current_bytes + data_len > self.max_bytes && !self.slots.is_empty())
{
// Find LRU slot
let lru_idx = self
.slots
.iter()
.enumerate()
.min_by_key(|(_, s)| s.last_access)
.map(|(i, _)| i)
.unwrap();
let removed = self.slots.swap_remove(lru_idx);
self.slot_index.remove(&removed.key);
// swap_remove moved the former last element into `lru_idx` (unless
// it *was* the last element) — fix up that element's index entry.
if lru_idx < self.slots.len() {
let moved_key = self.slots[lru_idx].key.clone();
self.slot_index.insert(moved_key, lru_idx);
}
self.current_bytes -= removed.data.len();
self.stats.evictions += 1;
}
self.current_bytes += data_len;
let new_idx = self.slots.len();
self.slot_index.insert(key.clone(), new_idx);
self.slots.push(CachedChunk {
key,
data: Arc::clone(&data),
last_access: tick,
});
data
}
}
/// Access pattern statistics tracked by the chunk cache.
@@ -356,8 +524,8 @@ impl ChunkCache {
pub fn with_capacity(max_bytes: usize, max_slots: usize) -> Self {
Self {
inner: std::sync::Mutex::new(CacheInner {
index: None,
index_addr: None,
datasets: HashMap::new(),
current: None,
slots: Vec::with_capacity(max_slots.min(64)),
slot_index: HashMap::with_capacity(max_slots.min(64)),
current_bytes: 0,
@@ -366,335 +534,331 @@ impl ChunkCache {
tick: 0,
last_coord: None,
stats: AccessStats::default(),
chunk_index: None,
chunk_layout: None,
}),
}
}
// ----- Index operations -----
fn lock(&self) -> std::sync::MutexGuard<'_, CacheInner> {
self.inner.lock().unwrap_or_else(|e| e.into_inner())
}
/// Bind the cache to the dataset at chunk-index address `addr`.
/// The most decompressed bytes this cache will hold.
pub fn max_bytes(&self) -> usize {
self.lock().max_bytes
}
// ----- Dataset-keyed operations (safe to use concurrently) -----
/// The chunk list of the dataset whose chunk index is at `addr`.
///
/// The cache is shared per file across all of its datasets. If the cache
/// currently holds state for a different dataset, all per-dataset state
/// (chunk index, chunk-index map, layout, and decompressed slots) is
/// dropped so the next access rebuilds it for this dataset. Reading the
/// same dataset again is a no-op, preserving the cache's benefit for
/// repeated/sequential access. Returns `true` if a reset occurred.
/// On the first call for a dataset, `build` scans its chunk index; the
/// result is kept (offsets truncated to `rank` for the lookup key), so
/// later calls skip the scan. `build` runs without the cache lock held;
/// if two threads race to build the same dataset's index, the first
/// stored one wins and both return equivalent lists.
pub fn chunks_for<E>(
&self,
addr: u64,
rank: usize,
build: impl FnOnce() -> Result<Vec<ChunkInfo>, E>,
) -> Result<Vec<ChunkInfo>, E> {
Ok(self
.index_for(addr, rank, build)?
.values()
.cloned()
.collect())
}
fn index_for<E>(
&self,
addr: u64,
rank: usize,
build: impl FnOnce() -> Result<Vec<ChunkInfo>, E>,
) -> Result<Arc<HashMap<ChunkCoord, ChunkInfo>>, E> {
if let Some(index) = self.lock().touch(addr).index.clone() {
return Ok(index);
}
let chunks = build()?;
let map: HashMap<ChunkCoord, ChunkInfo> = chunks
.into_iter()
.map(|ci| (ci.offsets.iter().take(rank).copied().collect(), ci))
.collect();
let mut inner = self.lock();
let entry = inner.touch(addr);
let index = Arc::clone(entry.index.get_or_insert_with(|| Arc::new(map)));
inner.trim_datasets(addr);
Ok(index)
}
/// The pre-computed assembly layout of the dataset at `addr`, building
/// its chunk index (via `build`, as in [`Self::chunks_for`]) and layout on
/// first use.
pub fn chunk_layout_for<E>(
&self,
addr: u64,
rank: usize,
build: impl FnOnce() -> Result<Vec<ChunkInfo>, E>,
ds_dims: &[usize],
chunk_dims: &[usize],
elem_size: usize,
) -> Result<Arc<ChunkLayout>, E> {
let (layout, chunk_index) = {
let mut inner = self.lock();
let entry = inner.touch(addr);
(entry.chunk_layout.clone(), entry.chunk_index.clone())
};
if let Some(layout) = layout {
return Ok(layout);
}
let chunk_index = match chunk_index {
Some(ci) => ci,
None => {
let index = self.index_for(addr, rank, build)?;
let chunks: Vec<ChunkInfo> = index.values().cloned().collect();
Arc::new(ChunkIndex::build(&chunks, rank))
}
};
let layout = ChunkLayout::build(&chunk_index, ds_dims, chunk_dims, elem_size);
let mut inner = self.lock();
let entry = inner.touch(addr);
entry.chunk_index.get_or_insert(chunk_index);
let layout = Arc::clone(entry.chunk_layout.get_or_insert_with(|| Arc::new(layout)));
inner.trim_datasets(addr);
Ok(layout)
}
/// Cached decompressed chunk at `coord` of the dataset at `addr`.
///
/// O(1) lookup; the clone is an `Arc` refcount bump, not a copy of the
/// underlying decompressed data.
pub fn get_decompressed_in(&self, addr: u64, coord: &[u64]) -> Option<Arc<CacheAlignedBuffer>> {
self.lock().get_decompressed(addr, coord)
}
/// Cache decompressed chunk data for `coord` of the dataset at `addr`.
/// Returns the `Arc`-shared buffer now cached (or already cached).
pub fn put_decompressed_in(
&self,
addr: u64,
coord: ChunkCoord,
data: Vec<u8>,
) -> Arc<CacheAlignedBuffer> {
self.put_decompressed_aligned_in(addr, coord, CacheAlignedBuffer::from_vec(data))
}
/// [`Self::put_decompressed_in`] for an already-aligned buffer.
pub fn put_decompressed_aligned_in(
&self,
addr: u64,
coord: ChunkCoord,
data: CacheAlignedBuffer,
) -> Arc<CacheAlignedBuffer> {
let data = Arc::new(data);
self.lock().put_decompressed((addr, coord), data)
}
/// Record that the given chunk coordinates of the dataset at `addr` are
/// predicted to be accessed soon (bookkeeping only).
///
/// This does **not** prefetch or pre-decompress anything — it only
/// checks whether each coordinate is already in the chunk index and
/// updates access-pattern stats accordingly.
pub fn prefetch_hint_in(&self, addr: u64, next_coords: &[ChunkCoord]) {
let mut inner = self.lock();
let Some(index) = inner.entry(addr).and_then(|e| e.index.clone()) else {
return;
};
let known = next_coords
.iter()
.filter(|c| index.contains_key(*c))
.count();
inner.stats.sequential_count += known as u64;
}
// ----- Address-less operations on the bound dataset -----
/// Bind the address-less methods to the dataset at chunk-index address
/// `addr`. Returns `true` if this changed the bound dataset.
///
/// Each dataset's state is kept separately, so switching loses nothing
/// and never exposes one dataset's index or chunks to another. The
/// binding itself is shared, though: concurrent readers should use the
/// `addr`-taking methods rather than bind and then call these.
pub fn ensure_dataset(&self, addr: u64) -> bool {
let mut inner = self.inner.lock().unwrap_or_else(|e| e.into_inner());
if inner.index_addr == Some(addr) {
return false;
}
inner.index = None;
inner.chunk_index = None;
inner.chunk_layout = None;
inner.slots.clear();
inner.slot_index.clear();
inner.current_bytes = 0;
inner.last_coord = None;
inner.index_addr = Some(addr);
true
let mut inner = self.lock();
let changed = inner.current != Some(addr);
inner.current = Some(addr);
changed
}
/// Returns `true` if the chunk index has been built.
/// Returns `true` if the bound dataset's chunk index has been built.
pub fn has_index(&self) -> bool {
self.inner
.lock()
.unwrap_or_else(|e| e.into_inner())
.index
.is_some()
let inner = self.lock();
inner
.entry(inner.current())
.is_some_and(|e| e.index.is_some())
}
/// Build the chunk index from a pre-collected list of `ChunkInfo`.
/// Build the bound dataset's chunk index from a pre-collected list of
/// `ChunkInfo`.
///
/// The `rank` parameter is used to truncate offsets to spatial dims only
/// (B-tree v1 stores rank+1 offsets).
pub fn populate_index(&self, chunks: &[ChunkInfo], rank: usize) {
let mut inner = self.inner.lock().unwrap_or_else(|e| e.into_inner());
if inner.index.is_some() {
return; // already populated
}
let mut map = HashMap::with_capacity(chunks.len());
for ci in chunks {
let coord: ChunkCoord = ci.offsets.iter().take(rank).copied().collect();
map.insert(coord, ci.clone());
}
inner.index = Some(map);
let addr = self.lock().current();
let _ = self.index_for::<core::convert::Infallible>(addr, rank, || Ok(chunks.to_vec()));
}
/// Look up a chunk by its spatial coordinate in the index.
/// Look up a chunk by its spatial coordinate in the bound dataset's index.
pub fn lookup_index(&self, coord: &[u64]) -> Option<ChunkInfo> {
let inner = self.inner.lock().unwrap_or_else(|e| e.into_inner());
inner.index.as_ref()?.get(coord).cloned()
let inner = self.lock();
inner
.entry(inner.current())?
.index
.as_ref()?
.get(coord)
.cloned()
}
/// Return all indexed chunks as a `Vec<ChunkInfo>` (order unspecified).
/// Return all of the bound dataset's indexed chunks (order unspecified).
pub fn all_indexed_chunks(&self) -> Option<Vec<ChunkInfo>> {
let inner = self.inner.lock().unwrap_or_else(|e| e.into_inner());
inner.index.as_ref().map(|m| m.values().cloned().collect())
let inner = self.lock();
let index = inner.entry(inner.current())?.index.as_ref()?;
Some(index.values().cloned().collect())
}
// ----- Chunk index (pre-built coordinate → ChunkInfo map) -----
/// Returns `true` if the chunk B-tree index has been built.
/// Returns `true` if the bound dataset's `ChunkIndex` has been built.
pub fn has_chunk_index(&self) -> bool {
self.inner
.lock()
.unwrap_or_else(|e| e.into_inner())
.chunk_index
.is_some()
let inner = self.lock();
inner
.entry(inner.current())
.is_some_and(|e| e.chunk_index.is_some())
}
/// Build and store the chunk B-tree index from a pre-collected list of `ChunkInfo`.
/// Build and store the bound dataset's `ChunkIndex`.
pub fn populate_chunk_index(&self, chunks: &[ChunkInfo], rank: usize) {
let mut inner = self.inner.lock().unwrap_or_else(|e| e.into_inner());
if inner.chunk_index.is_some() {
return;
}
inner.chunk_index = Some(ChunkIndex::build(chunks, rank));
let built = Arc::new(ChunkIndex::build(chunks, rank));
let mut inner = self.lock();
let addr = inner.current();
inner.touch(addr).chunk_index.get_or_insert(built);
inner.trim_datasets(addr);
}
// ----- Chunk layout (pre-computed assembly plan) -----
/// Returns `true` if the chunk layout has been computed.
/// Returns `true` if the bound dataset's chunk layout has been computed.
pub fn has_chunk_layout(&self) -> bool {
self.inner
.lock()
.unwrap_or_else(|e| e.into_inner())
.chunk_layout
.is_some()
let inner = self.lock();
inner
.entry(inner.current())
.is_some_and(|e| e.chunk_layout.is_some())
}
/// Build and store the pre-computed chunk layout for fast assembly.
/// Build and store the bound dataset's chunk layout (needs its
/// `ChunkIndex`; does nothing without one).
pub fn populate_chunk_layout(&self, ds_dims: &[usize], chunk_dims: &[usize], elem_size: usize) {
let mut inner = self.inner.lock().unwrap_or_else(|e| e.into_inner());
if inner.chunk_layout.is_some() {
let mut inner = self.lock();
let addr = inner.current();
let entry = inner.touch(addr);
if entry.chunk_layout.is_some() {
return;
}
if let Some(ref idx) = inner.chunk_index {
inner.chunk_layout = Some(ChunkLayout::build(idx, ds_dims, chunk_dims, elem_size));
if let Some(idx) = entry.chunk_index.clone() {
entry.chunk_layout = Some(Arc::new(ChunkLayout::build(
&idx, ds_dims, chunk_dims, elem_size,
)));
}
}
/// Execute a function with a reference to the chunk layout.
///
/// Returns `None` if the layout hasn't been computed yet.
/// Execute a function with a reference to the bound dataset's chunk
/// layout. Returns `None` if the layout hasn't been computed yet.
pub fn with_chunk_layout<F, R>(&self, f: F) -> Option<R>
where
F: FnOnce(&ChunkLayout) -> R,
{
let inner = self.inner.lock().unwrap_or_else(|e| e.into_inner());
inner.chunk_layout.as_ref().map(f)
let layout = {
let inner = self.lock();
inner.entry(inner.current())?.chunk_layout.clone()?
};
Some(f(&layout))
}
// ----- Decompressed data cache (LRU) -----
/// Try to get cached decompressed data for a chunk coordinate.
/// Try to get cached decompressed data for a chunk of the bound dataset.
///
/// O(1) lookup. Returns an owned copy for API compatibility with callers
/// that need a `Vec<u8>`; prefer [`Self::get_decompressed_aligned`] when
/// an `Arc`-shared buffer works for the caller, since that avoids the
/// copy entirely.
/// Returns an owned copy; prefer [`Self::get_decompressed_aligned`] when
/// an `Arc`-shared buffer works for the caller.
pub fn get_decompressed(&self, coord: &[u64]) -> Option<Vec<u8>> {
self.get_decompressed_aligned(coord)
.map(|arc| arc.as_slice().to_vec())
}
/// Try to get a reference-counted clone of the aligned buffer for a chunk.
///
/// O(1) index lookup; the clone is an `Arc` refcount bump, not a copy of
/// the underlying decompressed data.
/// Reference-counted cached buffer for a chunk of the bound dataset.
pub fn get_decompressed_aligned(&self, coord: &[u64]) -> Option<Arc<CacheAlignedBuffer>> {
let mut inner = self.inner.lock().unwrap_or_else(|e| e.into_inner());
inner.tick += 1;
let tick = inner.tick;
// Track sequential vs random access
let is_sequential = inner.last_coord.as_ref().is_some_and(|prev| {
// Sequential if exactly one dimension changed
let changes: usize = prev
.iter()
.zip(coord.iter())
.filter(|(a, b)| a != b)
.count();
changes <= 1
});
if is_sequential {
inner.stats.sequential_count += 1;
} else if inner.last_coord.is_some() {
inner.stats.random_count += 1;
}
inner.last_coord = Some(coord.to_vec());
let found = if let Some(&idx) = inner.slot_index.get(coord) {
inner.slots[idx].last_access = tick;
Some(Arc::clone(&inner.slots[idx].data))
} else {
None
};
if let Some(ref data) = found {
inner.stats.hits += 1;
inner.stats.bytes_read += data.len() as u64;
} else {
inner.stats.misses += 1;
}
found
let mut inner = self.lock();
let addr = inner.current();
inner.get_decompressed(addr, coord)
}
/// Insert decompressed chunk data into the LRU cache.
///
/// The data is stored in a [`CacheAlignedBuffer`] so subsequent reads
/// return cache-line-aligned memory. Returns the `Arc`-shared buffer that
/// is now cached (or already was), so the caller can reuse it directly
/// instead of holding a separate copy of the same data.
/// Insert decompressed chunk data for the bound dataset into the LRU
/// cache, returning the `Arc`-shared buffer now cached.
pub fn put_decompressed(&self, coord: ChunkCoord, data: Vec<u8>) -> Arc<CacheAlignedBuffer> {
let aligned = CacheAlignedBuffer::from_vec(data);
self.put_decompressed_aligned(coord, aligned)
self.put_decompressed_aligned(coord, CacheAlignedBuffer::from_vec(data))
}
/// Insert an already-aligned buffer into the LRU cache.
///
/// Returns the `Arc`-shared buffer now held by the cache (the one just
/// inserted, or the existing cached copy if `coord` was already present).
/// Insert an already-aligned buffer for the bound dataset.
pub fn put_decompressed_aligned(
&self,
coord: ChunkCoord,
data: CacheAlignedBuffer,
) -> Arc<CacheAlignedBuffer> {
let data = Arc::new(data);
let mut inner = self.inner.lock().unwrap_or_else(|e| e.into_inner());
let data_len = data.len();
// Don't cache if single chunk exceeds budget — still return the data
// to the caller, just don't retain it.
if data_len > inner.max_bytes {
return data;
let mut inner = self.lock();
let addr = inner.current();
inner.put_decompressed((addr, coord), data)
}
// Check if already present
inner.tick += 1;
let tick = inner.tick;
if let Some(&idx) = inner.slot_index.get(&coord) {
inner.slots[idx].last_access = tick;
return Arc::clone(&inner.slots[idx].data); // already cached
/// [`Self::prefetch_hint_in`] for the bound dataset.
pub fn prefetch_hint(&self, next_coords: &[ChunkCoord]) {
let addr = self.lock().current();
self.prefetch_hint_in(addr, next_coords);
}
// Evict until we have room
while inner.slots.len() >= inner.max_slots
|| (inner.current_bytes + data_len > inner.max_bytes && !inner.slots.is_empty())
{
// Find LRU slot
let lru_idx = inner
.slots
.iter()
.enumerate()
.min_by_key(|(_, s)| s.last_access)
.map(|(i, _)| i)
.unwrap();
let removed = inner.slots.swap_remove(lru_idx);
inner.slot_index.remove(&removed.coord);
// swap_remove moved the former last element into `lru_idx` (unless
// it *was* the last element) — fix up that element's index entry.
if lru_idx < inner.slots.len() {
let moved_coord = inner.slots[lru_idx].coord.clone();
inner.slot_index.insert(moved_coord, lru_idx);
}
inner.current_bytes -= removed.data.len();
inner.stats.evictions += 1;
}
// ----- Whole-cache operations -----
inner.current_bytes += data_len;
let new_idx = inner.slots.len();
inner.slot_index.insert(coord.clone(), new_idx);
inner.slots.push(CachedChunk {
coord,
data: Arc::clone(&data),
last_access: tick,
});
data
}
/// Clear the entire cache (index + decompressed data).
/// Clear the entire cache (indexes + decompressed data + stats).
pub fn clear(&self) {
let mut inner = self.inner.lock().unwrap_or_else(|e| e.into_inner());
inner.index = None;
inner.index_addr = None;
let mut inner = self.lock();
inner.datasets.clear();
inner.current = None;
inner.slots.clear();
inner.slot_index.clear();
inner.current_bytes = 0;
inner.tick = 0;
inner.last_coord = None;
inner.stats = AccessStats::default();
inner.chunk_index = None;
inner.chunk_layout = None;
}
/// Record that the given chunk coordinates are predicted to be accessed
/// soon (bookkeeping only).
///
/// This does **not** prefetch or pre-decompress anything — it only
/// checks whether each coordinate is already in the chunk index and
/// updates access-pattern stats accordingly. Real prefetching (e.g.
/// background pre-decompression) is not implemented.
pub fn prefetch_hint(&self, next_coords: &[ChunkCoord]) {
let inner = self.inner.lock().unwrap_or_else(|e| e.into_inner());
if inner.index.is_none() {
return;
}
drop(inner);
// For each predicted coordinate, verify it exists in the index.
// The index is already populated, so this is a no-op for known chunks.
// The purpose is to signal intent — callers can pre-decompress if needed.
// We touch the stats to record that prefetch hints were issued.
let mut inner = self.inner.lock().unwrap_or_else(|e| e.into_inner());
for coord in next_coords {
let exists = inner
.index
.as_ref()
.map(|idx| idx.contains_key(coord))
.unwrap_or(false);
if exists {
inner.stats.sequential_count += 1;
}
}
}
/// Return the current access pattern statistics.
pub fn access_stats(&self) -> AccessStats {
self.inner
.lock()
.unwrap_or_else(|e| e.into_inner())
.stats
.clone()
self.lock().stats.clone()
}
/// Update the sweep direction label in the access stats.
pub fn set_sweep_direction(&self, direction: &'static str) {
self.inner
.lock()
.unwrap_or_else(|e| e.into_inner())
.stats
.sweep_direction = Some(direction);
self.lock().stats.sweep_direction = Some(direction);
}
/// Number of decompressed chunks currently cached.
/// Number of decompressed chunks currently cached (all datasets).
pub fn cached_chunk_count(&self) -> usize {
self.inner
.lock()
.unwrap_or_else(|e| e.into_inner())
.slots
.len()
self.lock().slots.len()
}
/// Total bytes of decompressed data currently cached.
/// Total bytes of decompressed data currently cached (all datasets).
pub fn cached_bytes(&self) -> usize {
self.inner
.lock()
.unwrap_or_else(|e| e.into_inner())
.current_bytes
self.lock().current_bytes
}
/// Number of datasets whose chunk index is currently kept.
pub fn indexed_dataset_count(&self) -> usize {
self.lock().datasets.len()
}
}
@@ -803,6 +967,92 @@ mod tests {
assert_eq!(cache.cached_bytes(), 0);
}
#[test]
fn datasets_sharing_coordinates_stay_separate() {
let cache = ChunkCache::new();
let a = vec![make_chunk(vec![0, 0], 0x100, 8)];
let b = vec![make_chunk(vec![0, 0], 0x900, 8)];
let got_a = cache.chunks_for::<()>(1, 1, || Ok(a.clone())).unwrap();
let got_b = cache.chunks_for::<()>(2, 1, || Ok(b.clone())).unwrap();
assert_eq!(got_a[0].address, 0x100);
assert_eq!(got_b[0].address, 0x900);
// Built once per dataset: a second lookup doesn't call the builder.
let again = cache
.chunks_for::<()>(1, 1, || panic!("index rebuilt"))
.unwrap();
assert_eq!(again[0].address, 0x100);
cache.put_decompressed_in(1, vec![0], vec![1; 4]);
cache.put_decompressed_in(2, vec![0], vec![2; 4]);
assert_eq!(
cache.get_decompressed_in(1, &[0]).unwrap().as_slice(),
&[1; 4]
);
assert_eq!(
cache.get_decompressed_in(2, &[0]).unwrap().as_slice(),
&[2; 4]
);
assert!(cache.get_decompressed_in(3, &[0]).is_none());
assert_eq!(cache.cached_chunk_count(), 2);
// The bound-dataset methods see only the bound dataset.
cache.ensure_dataset(2);
assert_eq!(cache.lookup_index(&[0]).unwrap().address, 0x900);
assert_eq!(cache.get_decompressed(&[0]).unwrap(), vec![2; 4]);
}
#[test]
fn dataset_indexes_are_bounded() {
let cache = ChunkCache::new();
for addr in 0..(MAX_INDEXED_DATASETS as u64 + 10) {
cache
.chunks_for::<()>(addr, 1, || Ok(vec![make_chunk(vec![0], addr, 8)]))
.unwrap();
}
assert_eq!(cache.indexed_dataset_count(), MAX_INDEXED_DATASETS);
// One huge index evicts the others but is itself kept.
let huge: Vec<ChunkInfo> = (0..MAX_INDEXED_CHUNKS as u64)
.map(|i| make_chunk(vec![i], i, 8))
.collect();
let got = cache.chunks_for::<()>(9999, 1, || Ok(huge)).unwrap();
assert_eq!(got.len(), MAX_INDEXED_CHUNKS);
assert_eq!(cache.indexed_dataset_count(), 1);
}
#[test]
fn concurrent_readers_of_different_datasets_see_their_own_chunks() {
let cache = std::sync::Arc::new(ChunkCache::with_capacity(1 << 20, 64));
let handles: Vec<_> = (0..8u64)
.map(|t| {
let cache = std::sync::Arc::clone(&cache);
std::thread::spawn(move || {
for round in 0..500u64 {
let addr = (t + round) % 16;
let coord = vec![round % 4];
let chunks = cache
.chunks_for::<()>(addr, 1, || {
Ok((0..4).map(|c| make_chunk(vec![c], addr, 8)).collect())
})
.unwrap();
assert!(chunks.iter().all(|c| c.address == addr));
let want = vec![addr as u8; 8];
let got = match cache.get_decompressed_in(addr, &coord) {
Some(hit) => hit.to_vec(),
None => cache
.put_decompressed_in(addr, coord, want.clone())
.to_vec(),
};
assert_eq!(got, want);
}
})
})
.collect();
for h in handles {
h.join().unwrap();
}
}
#[test]
fn duplicate_insert_is_noop() {
let cache = ChunkCache::new();
crates/clawhdf5-format/src/chunk_grid.rs
+200
View File
@@ -0,0 +1,200 @@
//! Chunk-index linearisation shared by the Fixed Array and Extensible Array
//! chunk indexes (reader and writer).
//!
//! Both indexes store one element per chunk at a *linear* index, and the
//! library derives that index from the chunk's scaled coordinates
//! (`offset / chunk_dim`) using the dataset's **maximum** dimensions, not its
//! current ones (`H5D__farray_idx_get_addr` / `H5D__earray_idx_get_addr`,
//! via `layout->max_down_chunks`). A dataset whose current shape is smaller
//! than its maxshape therefore has gaps in the index, and laying it out by the
//! current shape puts every chunk after the first row in the wrong place.
//!
//! The Extensible Array adds one more step: its one unlimited dimension has no
//! finite chunk count, so the library *swizzles* the coordinates to make that
//! dimension the slowest-varying one (`H5VM_swizzle_coords`, which moves
//! `coords[unlim_dim]` to the front and shifts the dimensions before it right
//! by one) before linearising with `swizzled_max_down_chunks`. When the
//! unlimited dimension is already dimension 0 no swizzle happens.
#[cfg(not(feature = "std"))]
extern crate alloc;
#[cfg(not(feature = "std"))]
use alloc::{vec, vec::Vec};
use crate::error::FormatError;
/// How a chunk index maps linear element indexes to chunk coordinates.
#[derive(Debug, Clone)]
pub(crate) struct ChunkGrid {
/// Spatial chunk dimensions, in dataset order.
chunk_dims: Vec<u64>,
/// Chunks per dimension covering the *current* extent, in dataset order.
cur_chunks: Vec<u64>,
/// Dataset dimension stored at each linearisation position (slowest
/// first). The identity except for a swizzled Extensible Array.
order: Vec<usize>,
/// Linear stride of each linearisation position.
down: Vec<u64>,
}
impl ChunkGrid {
/// Grid for a Fixed Array index: row-major over the chunk counts of the
/// maximum dimensions (`max_dims`, falling back to the current dimensions
/// when the dataspace records none).
pub(crate) fn fixed_array(
cur_dims: &[u64],
max_dims: Option<&[u64]>,
chunk_dims: &[u64],
) -> Result<Self, FormatError> {
Self::build(cur_dims, max_dims, chunk_dims, None)
}
/// Grid for an Extensible Array index: like the Fixed Array, but the
/// unlimited dimension (the one whose maximum is `H5S_UNLIMITED`) is moved
/// to the slowest-varying position first.
pub(crate) fn extensible_array(
cur_dims: &[u64],
max_dims: Option<&[u64]>,
chunk_dims: &[u64],
) -> Result<Self, FormatError> {
let unlim = max_dims.and_then(|m| m.iter().position(|&d| d == u64::MAX));
Self::build(cur_dims, max_dims, chunk_dims, unlim)
}
fn build(
cur_dims: &[u64],
max_dims: Option<&[u64]>,
chunk_dims: &[u64],
unlim: Option<usize>,
) -> Result<Self, FormatError> {
let rank = chunk_dims.len();
if cur_dims.len() != rank || max_dims.is_some_and(|m| m.len() != rank) {
return Err(FormatError::ChunkedReadError(
"chunk index rank does not match the dataspace".into(),
));
}
if chunk_dims.contains(&0) {
return Err(FormatError::ChunkedReadError(
"chunk dimension is zero".into(),
));
}
let cur_chunks: Vec<u64> = cur_dims
.iter()
.zip(chunk_dims)
.map(|(&d, &c)| d.div_ceil(c))
.collect();
// Chunk counts of the maximum extent. An unlimited dimension has no
// finite count; it only ever sits in the slowest position, where its
// count never enters a stride. A (corrupt) maximum smaller than the
// current extent is widened so no allocated chunk becomes unreachable.
let max_chunks: Vec<u64> = (0..rank)
.map(|d| {
let max = max_dims.map_or(cur_dims[d], |m| m[d]);
if max == u64::MAX {
u64::MAX
} else {
max.div_ceil(chunk_dims[d]).max(cur_chunks[d])
}
})
.collect();
let mut order: Vec<usize> = (0..rank).collect();
if let Some(u) = unlim {
order.remove(u);
order.insert(0, u);
}
let mut down = vec![1u64; rank];
for p in (0..rank.saturating_sub(1)).rev() {
let next = max_chunks[order[p + 1]];
if next == u64::MAX {
// Only reachable with more than one unlimited dimension, which
// neither index type can describe.
return Err(FormatError::ChunkedReadError(
"array chunk index with more than one unlimited dimension".into(),
));
}
down[p] = down[p + 1].checked_mul(next).ok_or_else(|| {
FormatError::Overflow("chunk index linear stride overflows u64".into())
})?;
}
Ok(Self {
chunk_dims: chunk_dims.to_vec(),
cur_chunks,
order,
down,
})
}
/// Dataset-space offsets of the chunk stored at linear `index`, or `None`
/// when that chunk lies outside the current extent (the index still has a
/// slot for it; the library ignores such chunks on read).
pub(crate) fn offsets(&self, index: u64) -> Option<Vec<u64>> {
let rank = self.chunk_dims.len();
let mut offsets = vec![0u64; rank];
let mut rem = index;
for p in 0..rank {
let d = self.order[p];
let scaled = rem / self.down[p];
rem %= self.down[p];
if scaled >= self.cur_chunks[d] {
return None;
}
offsets[d] = scaled * self.chunk_dims[d];
}
Some(offsets)
}
/// Linear index of the chunk with scaled coordinates `scaled`
/// (`offset / chunk_dim` per dimension, in dataset order).
pub(crate) fn linear_index(&self, scaled: &[u64]) -> u64 {
self.order
.iter()
.zip(&self.down)
.map(|(&d, &stride)| scaled[d] * stride)
.sum()
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn fixed_array_uses_max_dims() {
// shape (4, 6), chunks (2, 3), maxshape (20, 10): 10 x 4 chunk grid.
let g = ChunkGrid::fixed_array(&[4, 6], Some(&[20, 10]), &[2, 3]).unwrap();
assert_eq!(g.offsets(0), Some(vec![0, 0]));
assert_eq!(g.offsets(1), Some(vec![0, 3]));
assert_eq!(g.offsets(2), None); // column chunk 2 is beyond the extent
assert_eq!(g.offsets(4), Some(vec![2, 0]));
assert_eq!(g.offsets(5), Some(vec![2, 3]));
assert_eq!(g.offsets(8), None); // row chunk 2 is beyond the extent
assert_eq!(g.linear_index(&[1, 1]), 5);
}
#[test]
fn extensible_array_swizzles_unlimited_dim() {
// maxshape (10, None): dim 1 is unlimited and becomes slowest.
let g = ChunkGrid::extensible_array(&[4, 6], Some(&[10, u64::MAX]), &[2, 3]).unwrap();
// max chunks of dim 0 = 5, so index = c1 * 5 + c0.
assert_eq!(g.linear_index(&[1, 0]), 1);
assert_eq!(g.linear_index(&[0, 1]), 5);
assert_eq!(g.offsets(5), Some(vec![0, 3]));
assert_eq!(g.offsets(6), Some(vec![2, 3]));
assert_eq!(g.offsets(2), None);
}
#[test]
fn extensible_array_unlimited_first_is_row_major() {
let g = ChunkGrid::extensible_array(&[4, 6], Some(&[u64::MAX, 30]), &[2, 3]).unwrap();
// max chunks of dim 1 = 10.
assert_eq!(g.linear_index(&[1, 1]), 11);
assert_eq!(g.offsets(11), Some(vec![2, 3]));
}
#[test]
fn rejects_two_unlimited_dims_after_the_first() {
assert!(ChunkGrid::fixed_array(&[4, 6], Some(&[u64::MAX, u64::MAX]), &[2, 3]).is_err());
}
}
crates/clawhdf5-format/src/chunk_index.rs
+10 -1
View File
@@ -18,6 +18,7 @@ use alloc::collections::BTreeMap;
#[cfg(feature = "std")]
use std::collections::HashMap;
use crate::addr::to_usize;
use crate::chunk_cache::ChunkCoord;
use crate::chunked_read::ChunkInfo;
@@ -167,7 +168,15 @@ impl ChunkLayout {
for (_coord, ci) in index.iter() {
let coord: ChunkCoord = ci.offsets.iter().take(rank).copied().collect();
let chunk_offsets: Vec<usize> = coord.iter().map(|&o| o as usize).collect();
// `ds_dims` are `usize`: a chunk at an offset past `usize::MAX`
// (only on a 32-bit target) lies outside the dataset.
let Ok(chunk_offsets) = coord
.iter()
.map(|&o| to_usize(o))
.collect::<Result<Vec<usize>, _>>()
else {
continue;
};
let copies = if rank == 0 {
// Scalar dataset — single copy
crates/clawhdf5-format/src/chunked_read.rs
+1978 -742
View File
File diff suppressed because it is too large Load Diff
crates/clawhdf5-format/src/chunked_write.rs
+837 -195
View File
File diff suppressed because it is too large Load Diff
crates/clawhdf5-format/src/data_layout.rs
+546 -54
View File
@@ -1,12 +1,14 @@
//! HDF5 Data Layout message parsing (message type 0x0008).
#[cfg(not(feature = "std"))]
use alloc::{string::String, vec::Vec};
use alloc::{format, string::String, vec::Vec};
#[cfg(feature = "std")]
use std::string::String;
use crate::addr::to_usize;
use crate::error::FormatError;
use crate::storage::Storage;
/// A single VDS (Virtual Dataset) source mapping.
///
@@ -24,6 +26,34 @@ pub struct VdsMapping {
pub virtual_selection: Vec<u8>,
}
/// Most dimensions a layout message can list (libhdf5 `H5O_LAYOUT_NDIMS`):
/// 32 dataspace dimensions plus the element size.
const MAX_LAYOUT_NDIMS: usize = 33;
/// libhdf5's checks on a chunked layout message's dimensions
/// (`H5O__layout_decode`): at most [`MAX_LAYOUT_NDIMS`], no dimension 0, and
/// before version 4 at least one dataspace dimension plus the element size.
/// A zero chunk dimension used to read the dataset as all fill values.
fn check_chunk_dims(dims: Vec<u32>, layout_version: u8) -> Result<Vec<u32>, FormatError> {
if dims.len() > MAX_LAYOUT_NDIMS {
return Err(FormatError::InvalidChunkDimensions(
"dimensionality is too large".into(),
));
}
if layout_version < 4 && dims.len() < 2 {
return Err(FormatError::InvalidChunkDimensions(
"bad dimensions for chunked storage".into(),
));
}
if let Some(u) = dims.iter().position(|&d| d == 0) {
return Err(FormatError::InvalidChunkDimensions(format!(
"bad chunk dimension value when parsing layout message - chunk dimension must be \
positive: mesg->u.chunk.dim[{u}] = 0"
)));
}
Ok(dims)
}
/// Parsed HDF5 data layout message.
#[derive(Debug, Clone, PartialEq)]
pub enum DataLayout {
@@ -45,7 +75,9 @@ pub enum DataLayout {
chunk_dimensions: Vec<u32>,
/// B-tree address, or `None` if undefined.
btree_address: Option<u64>,
/// Layout version (3 or 4).
/// Layout version (3 or 4). Version 1/2 messages (HDF5 1.4/1.6-era)
/// use the same version-1 B-tree chunk index as version 3 and are
/// reported as 3.
version: u8,
/// Chunk index type (v4 only).
chunk_index_type: Option<u8>,
@@ -53,6 +85,11 @@ pub enum DataLayout {
single_chunk_filtered_size: Option<u64>,
/// Filter mask for v4 single chunk with filters.
single_chunk_filter_mask: Option<u32>,
/// Layout v4 flag bit 0 (`H5D_CHUNK_DONT_FILTER_PARTIAL_CHUNKS`):
/// partial edge chunks — those extending past the dataset's current
/// extent in some dimension — are stored without the filter pipeline,
/// even though their filter mask is 0. Always `false` for v3.
dont_filter_partial_edge_chunks: bool,
},
/// Virtual dataset layout (v4 only).
Virtual {
@@ -67,21 +104,33 @@ pub enum DataLayout {
},
}
/// Version-1 VDS mapping flag: the source file name is stored by an earlier
/// entry, whose index follows in place of the name.
const VDS_SOURCE_FILE_SHARED: u8 = 0x01;
/// Version-1 VDS mapping flag: likewise for the source dataset name.
const VDS_SOURCE_DSET_SHARED: u8 = 0x02;
/// Version-1 VDS mapping flag: the source is in the virtual file itself
/// (`"."`); no file name is stored.
const VDS_SOURCE_SAME_FILE: u8 = 0x04;
const VDS_ALL_FLAGS: u8 = VDS_SOURCE_FILE_SHARED | VDS_SOURCE_DSET_SHARED | VDS_SOURCE_SAME_FILE;
/// Parse VDS mappings from global-heap object data.
///
/// The global-heap block holding a VDS mapping list is laid out as
/// (reverse-engineered and validated against HDF5 2.0):
/// (`H5D__virtual_store_layout` / `H5D__virtual_load_layout` in libhdf5):
///
/// ```text
/// version(1) · nused(length_size, LE) · entry[nused] · checksum(4)
/// ```
///
/// Each entry is:
/// - source file name — a null-terminated string in **block version 0**; in
/// **block version 1** a same-file reference is encoded as a single `0x04`
/// marker byte (the source file is the virtual file itself) in place of the
/// name;
/// - source dataset name (null-terminated string);
/// - **block version 1 only:** a flags byte. `0x04`: the source is in the
/// virtual file itself and no file name is stored; `0x01`/`0x02`: the
/// source file/dataset name is that of an earlier entry, whose index
/// (`length_size` bytes) is stored instead of the name. libhdf5 2.0 writes
/// version 1 when the file's low version bound is 2.0 and it saves space;
/// - source file name (null-terminated string, unless flagged above);
/// - source dataset name (null-terminated string, unless flagged above);
/// - source selection (serialized `H5S` dataspace selection — self-describing
/// in length);
/// - virtual selection (serialized `H5S` dataspace selection).
@@ -107,7 +156,7 @@ pub fn parse_vds_mappings(
// `nused` is untrusted; don't pre-allocate from it. Each entry consumes at
// least a few bytes, so the loop is naturally bounded by the heap data and
// a bogus `nused` simply errors out on the first short read.
let mut mappings = Vec::new();
let mut mappings: Vec<VdsMapping> = Vec::new();
// Reads one self-describing selection at `pos`, returning its raw bytes and
// advancing past it — bounds-checked so a corrupt selection can't overrun.
let read_selection = |heap_data: &[u8], pos: &mut usize| -> Result<Vec<u8>, FormatError> {
@@ -127,17 +176,57 @@ pub fn parse_vds_mappings(
Ok(bytes)
};
for _ in 0..nused {
// Source file name (with the version-1 same-file marker handled).
let source_file = if version >= 1 && heap_data.get(pos) == Some(&0x04) {
if version > 1 {
return Err(FormatError::ChunkedReadError(
"unsupported VDS mapping block version".into(),
));
}
for i in 0..nused {
// Version 1 prefixes each entry with a flags byte; a name may then be
// omitted (same file) or replaced by the index of an earlier entry
// holding the same name (`H5D__virtual_load_layout`).
let flags = if version >= 1 {
let f = *heap_data.get(pos).ok_or(FormatError::UnexpectedEof {
expected: pos + 1,
available: heap_data.len(),
})?;
pos += 1;
if f & !VDS_ALL_FLAGS != 0 {
return Err(FormatError::ChunkedReadError(
"unknown VDS mapping flags".into(),
));
}
f
} else {
0
};
// Index of an earlier entry, for a shared name.
let earlier = |pos: &mut usize| -> Result<usize, FormatError> {
let idx = read_length(heap_data, *pos, length_size)?;
*pos += ls;
if idx >= i {
return Err(FormatError::ChunkedReadError(
"VDS mapping shares a name with a later entry".into(),
));
}
to_usize(idx)
};
let source_file = if flags & VDS_SOURCE_SAME_FILE != 0 {
String::from(".")
} else if flags & VDS_SOURCE_FILE_SHARED != 0 {
let idx = earlier(&mut pos)?;
mappings[idx].source_file.clone()
} else {
read_null_terminated_string(heap_data, &mut pos)?
};
// Source dataset name.
let source_dataset = read_null_terminated_string(heap_data, &mut pos)?;
let source_dataset = if flags & VDS_SOURCE_DSET_SHARED != 0 {
let idx = earlier(&mut pos)?;
mappings[idx].source_dataset.clone()
} else {
read_null_terminated_string(heap_data, &mut pos)?
};
// Source selection, then virtual selection (both self-describing length).
let source_selection = read_selection(heap_data, &mut pos)?;
@@ -222,6 +311,16 @@ impl DataLayout {
&mut self,
file_data: &[u8],
length_size: u8,
) -> Result<(), FormatError> {
self.resolve_vds_mappings_in(file_data, length_size)
}
/// [`Self::resolve_vds_mappings`] over any [`Storage`]: one read of the
/// global heap collection holding the mappings.
pub fn resolve_vds_mappings_in<S: Storage + ?Sized>(
&mut self,
file_data: &S,
length_size: u8,
) -> Result<(), FormatError> {
if let DataLayout::Virtual {
global_heap_address,
@@ -231,11 +330,8 @@ impl DataLayout {
} = self
&& let Some(addr) = *global_heap_address
{
let coll = crate::global_heap::GlobalHeapCollection::parse(
file_data,
addr as usize,
length_size,
)?;
let coll =
crate::global_heap::GlobalHeapCollection::parse_in(file_data, addr, length_size)?;
let obj = coll.get_object(*global_heap_index as u16).ok_or(
FormatError::GlobalHeapObjectNotFound {
collection_address: addr,
@@ -256,6 +352,7 @@ impl DataLayout {
let layout_class = data[1];
match version {
1 | 2 => Self::parse_v1_v2(data, offset_size),
3 => Self::parse_v3(data, layout_class, offset_size, length_size),
// v5 (emitted by HDF5 1.14+/2.0 with `libver=latest`) uses the same
// message structure as v4 — only the version number was bumped.
@@ -264,6 +361,87 @@ impl DataLayout {
}
}
/// Layout message versions 1 and 2 (HDF5 before 1.6.3):
///
/// ```text
/// version(1) · dimensionality(1) · layout class(1) · reserved(5)
/// · address(offset_size) — contiguous and chunked only
/// · dimension sizes(4 × dimensionality)
/// · compact data size(4) · compact raw data — compact only
/// ```
///
/// The dimension sizes are the dataset's (contiguous/compact) or the
/// chunk's (chunked) extent plus a trailing element-size dimension, as in
/// version 3's chunked form. libhdf5 ignores them for contiguous storage
/// and sizes the data from the dataspace; the product of the stored
/// dimensions is that same size, and a disagreement (a dimension that was
/// truncated to 32 bits) is caught by the reader's size check rather than
/// returning wrong data.
fn parse_v1_v2(data: &[u8], offset_size: u8) -> Result<DataLayout, FormatError> {
ensure_len(data, 0, 8)?;
let dimensionality = data[1] as usize;
let layout_class = data[2];
// H5O_LAYOUT_NDIMS: 32 dataspace dimensions + the element-size one.
if dimensionality > 33 {
return Err(FormatError::Overflow(format!(
"data layout dimensionality {dimensionality} exceeds 33"
)));
}
let mut p = 8;
let os = offset_size as usize;
let address = match layout_class {
1 | 2 => {
ensure_len(data, p, os)?;
let a = if is_undefined(data, p, offset_size) {
None
} else {
Some(read_offset(data, p, offset_size)?)
};
p += os;
a
}
0 => None,
_ => return Err(FormatError::InvalidLayoutClass(layout_class)),
};
ensure_len(data, p, dimensionality * 4)?;
let dims: Vec<u32> = data[p..p + dimensionality * 4]
.as_chunks::<4>()
.0
.iter()
.map(|c| u32::from_le_bytes(*c))
.collect();
p += dimensionality * 4;
match layout_class {
0 => {
ensure_len(data, p, 4)?;
let size =
u32::from_le_bytes([data[p], data[p + 1], data[p + 2], data[p + 3]]) as usize;
ensure_len(data, p + 4, size)?;
Ok(DataLayout::Compact {
data: data[p + 4..p + 4 + size].to_vec(),
})
}
1 => {
let size = dims
.iter()
.try_fold(1u64, |acc, &d| acc.checked_mul(d as u64))
.ok_or_else(|| {
FormatError::Overflow(format!("contiguous layout size {dims:?}"))
})?;
Ok(DataLayout::Contiguous { address, size })
}
_ => Ok(DataLayout::Chunked {
chunk_dimensions: check_chunk_dims(dims, 2)?,
btree_address: address,
version: 3,
chunk_index_type: None,
single_chunk_filtered_size: None,
single_chunk_filter_mask: None,
dont_filter_partial_edge_chunks: false,
}),
}
}
fn parse_v3(
data: &[u8],
layout_class: u8,
@@ -316,12 +494,13 @@ impl DataLayout {
p += 4;
}
Ok(DataLayout::Chunked {
chunk_dimensions,
chunk_dimensions: check_chunk_dims(chunk_dimensions, 3)?,
btree_address,
version: 3,
chunk_index_type: None,
single_chunk_filtered_size: None,
single_chunk_filter_mask: None,
dont_filter_partial_edge_chunks: false,
})
}
_ => Err(FormatError::InvalidLayoutClass(layout_class)),
@@ -364,47 +543,40 @@ impl DataLayout {
let dimensionality = data[pos + 1] as usize;
let dim_size_encoded_length = data[pos + 2] as usize;
let mut p = pos + 3;
if dimensionality > MAX_LAYOUT_NDIMS {
return Err(FormatError::InvalidChunkDimensions(
"dimensionality is too large".into(),
));
}
// dimension sizes
// Each dimension takes 1 to 8 bytes (libhdf5 writes the
// fewest that hold the largest one, so 3, 5, 6 and 7 occur:
// a chunk dimension of 70 000 takes 3). libhdf5 refuses 0
// and more than 8.
if dim_size_encoded_length == 0 || dim_size_encoded_length > 8 {
return Err(FormatError::InvalidChunkDimensions(
"encoded chunk dimension size is too large".into(),
));
}
ensure_len(data, p, dimensionality * dim_size_encoded_length)?;
let mut chunk_dimensions = Vec::with_capacity(dimensionality);
for _ in 0..dimensionality {
let val = match dim_size_encoded_length {
1 => data[p] as u32,
2 => u16::from_le_bytes([data[p], data[p + 1]]) as u32,
4 => u32::from_le_bytes([data[p], data[p + 1], data[p + 2], data[p + 3]]),
8 => {
// V4 chunked encodes dimension sizes as 8 bytes, but
// our ChunkedStorageV4 stores them as u32. We read only
// the low 4 bytes (little-endian). This silently
// truncates dimensions > 4 GiB, which are not expected
// in practice (HDF5 chunk dimensions are always small).
// If the high bytes are non-zero, the file is malformed
// or uses dimensions we cannot represent.
let high = u32::from_le_bytes([
data[p + 4],
data[p + 5],
data[p + 6],
data[p + 7],
]);
if high != 0 {
return Err(FormatError::UnexpectedEof {
expected: p + 8,
available: data.len(),
});
}
u32::from_le_bytes([data[p], data[p + 1], data[p + 2], data[p + 3]])
}
_ => {
return Err(FormatError::UnexpectedEof {
expected: p + dim_size_encoded_length,
available: data.len(),
});
}
};
let val = data[p..p + dim_size_encoded_length]
.iter()
.rev()
.fold(0u64, |acc, &b| (acc << 8) | u64::from(b));
// Chunk dimensions are held as u32; HDF5 2.0 can write
// larger ones (layout version 5), which are refused
// rather than truncated.
let val = u32::try_from(val).map_err(|_| {
FormatError::InvalidChunkDimensions(format!(
"chunk dimension {val} is larger than 2^32 - 1, which is not supported"
))
})?;
chunk_dimensions.push(val);
p += dim_size_encoded_length;
}
let chunk_dimensions = check_chunk_dims(chunk_dimensions, 4)?;
// chunk index type
ensure_len(data, p, 1)?;
@@ -505,6 +677,7 @@ impl DataLayout {
chunk_index_type: Some(chunk_index_type),
single_chunk_filtered_size,
single_chunk_filter_mask,
dont_filter_partial_edge_chunks: flags & 0x01 != 0,
})
}
3 => {
@@ -539,6 +712,202 @@ impl DataLayout {
mod tests {
use super::*;
/// Version 1/2 header: version, dimensionality, class, reserved(5).
fn v1v2_header(version: u8, ndims: u8, class: u8) -> Vec<u8> {
vec![version, ndims, class, 0, 0, 0, 0, 0]
}
#[test]
fn v2_compact() {
let mut buf = v1v2_header(2, 2, 0);
// dims (3 elements of 2 bytes) — no address for compact
buf.extend_from_slice(&3u32.to_le_bytes());
buf.extend_from_slice(&2u32.to_le_bytes());
buf.extend_from_slice(&6u32.to_le_bytes()); // compact size (u32 in v1/v2)
buf.extend_from_slice(&[1, 0, 2, 0, 3, 0]);
assert_eq!(
DataLayout::parse(&buf, 8, 8).unwrap(),
DataLayout::Compact {
data: vec![1, 0, 2, 0, 3, 0]
}
);
}
#[test]
fn v1_contiguous_size_from_dimensions() {
let mut buf = v1v2_header(1, 3, 1);
buf.extend_from_slice(&0x800u32.to_le_bytes()); // 4-byte address
for d in [10u32, 20, 4] {
buf.extend_from_slice(&d.to_le_bytes());
}
assert_eq!(
DataLayout::parse(&buf, 4, 4).unwrap(),
DataLayout::Contiguous {
address: Some(0x800),
size: 800,
}
);
}
#[test]
fn v1_contiguous_undefined_address() {
let mut buf = v1v2_header(1, 2, 1);
buf.extend_from_slice(&[0xFF; 8]);
buf.extend_from_slice(&5u32.to_le_bytes());
buf.extend_from_slice(&8u32.to_le_bytes());
assert_eq!(
DataLayout::parse(&buf, 8, 8).unwrap(),
DataLayout::Contiguous {
address: None,
size: 40,
}
);
}
#[test]
fn v1_chunked_maps_to_btree_v1_index() {
let mut buf = v1v2_header(1, 3, 2);
buf.extend_from_slice(&0x1234u64.to_le_bytes());
for d in [50u32, 50, 4] {
buf.extend_from_slice(&d.to_le_bytes());
}
assert_eq!(
DataLayout::parse(&buf, 8, 8).unwrap(),
DataLayout::Chunked {
chunk_dimensions: vec![50, 50, 4],
btree_address: Some(0x1234),
version: 3,
chunk_index_type: None,
single_chunk_filtered_size: None,
single_chunk_filter_mask: None,
dont_filter_partial_edge_chunks: false,
}
);
}
/// A v3 chunked layout message with these dims (element size last).
fn v3_chunked_msg(dims: &[u32]) -> Vec<u8> {
let mut buf = vec![3u8, 2, dims.len() as u8];
buf.extend_from_slice(&0x1000u64.to_le_bytes());
for d in dims {
buf.extend_from_slice(&d.to_le_bytes());
}
buf
}
#[test]
fn chunk_dimensions_are_checked_when_the_layout_is_parsed() {
assert!(DataLayout::parse(&v3_chunked_msg(&[4, 4, 8]), 8, 8).is_ok());
// A zero chunk dimension used to read as all fill values.
let err = DataLayout::parse(&v3_chunked_msg(&[4, 0, 8]), 8, 8).unwrap_err();
assert!(
matches!(&err, FormatError::InvalidChunkDimensions(m) if m.contains("dim[1] = 0")),
"{err:?}"
);
// Only the element-size dimension: libhdf5 "bad dimensions".
assert_eq!(
DataLayout::parse(&v3_chunked_msg(&[8]), 8, 8).unwrap_err(),
FormatError::InvalidChunkDimensions("bad dimensions for chunked storage".into())
);
assert_eq!(
DataLayout::parse(&v3_chunked_msg(&[1; 34]), 8, 8).unwrap_err(),
FormatError::InvalidChunkDimensions("dimensionality is too large".into())
);
// v1/v2 and v4 messages get the zero check too.
let mut v1 = v1v2_header(1, 2, 2);
v1.extend_from_slice(&0x1000u64.to_le_bytes());
v1.extend_from_slice(&0u32.to_le_bytes());
v1.extend_from_slice(&8u32.to_le_bytes());
assert!(matches!(
DataLayout::parse(&v1, 8, 8),
Err(FormatError::InvalidChunkDimensions(_))
));
let mut v4 = vec![4u8, 2, 0, 2, 4];
v4.extend_from_slice(&0u32.to_le_bytes());
v4.extend_from_slice(&8u32.to_le_bytes());
v4.push(3); // fixed array index
v4.push(0); // page bits
v4.extend_from_slice(&0x1000u64.to_le_bytes());
assert!(matches!(
DataLayout::parse(&v4, 8, 8),
Err(FormatError::InvalidChunkDimensions(_))
));
}
/// A v4 chunked layout (fixed array index) whose `dims` are each
/// encoded in `width` bytes.
fn v4_chunked_msg(width: u8, dims: &[u64]) -> Vec<u8> {
let mut m = vec![4u8, 2, 0, dims.len() as u8, width];
for &d in dims {
m.extend_from_slice(&d.to_le_bytes()[..width.min(8) as usize]);
}
m.push(3); // fixed array index
m.push(0); // page bits
m.extend_from_slice(&0x1000u64.to_le_bytes());
m
}
#[test]
fn v4_chunk_dimensions_take_1_to_8_bytes() {
// libhdf5 encodes each dimension in the fewest bytes that hold the
// largest: a chunk dimension of 70 000 takes 3, and 3, 5, 6 and 7
// were refused ("UnexpectedEof").
for width in 1..=8u8 {
let dims = [if width >= 3 { 70_000 } else { 200 }, 8];
let layout = DataLayout::parse(&v4_chunked_msg(width, &dims), 8, 8)
.unwrap_or_else(|e| panic!("width {width}: {e:?}"));
assert!(
matches!(&layout, DataLayout::Chunked { chunk_dimensions, .. }
if chunk_dimensions.iter().map(|&d| u64::from(d)).eq(dims)),
"width {width}: {layout:?}"
);
}
// libhdf5 refuses 0 and more than 8 bytes.
for width in [0u8, 9] {
assert_eq!(
DataLayout::parse(&v4_chunked_msg(width, &[4, 8]), 8, 8).unwrap_err(),
FormatError::InvalidChunkDimensions(
"encoded chunk dimension size is too large".into()
)
);
}
// A dimension past u32 cannot be represented and is refused, not
// truncated.
assert!(matches!(
DataLayout::parse(&v4_chunked_msg(5, &[1 << 32, 8]), 8, 8),
Err(FormatError::InvalidChunkDimensions(m)) if m.contains("2^32")
));
}
#[test]
fn v1v2_rejects_bad_class_dimensionality_and_truncation() {
assert_eq!(
DataLayout::parse(&v1v2_header(1, 1, 3), 8, 8).unwrap_err(),
FormatError::InvalidLayoutClass(3)
);
assert!(matches!(
DataLayout::parse(&v1v2_header(2, 34, 1), 8, 8).unwrap_err(),
FormatError::Overflow(_)
));
// Chunked, dims cut short.
let mut buf = v1v2_header(1, 2, 2);
buf.extend_from_slice(&0x10u64.to_le_bytes());
buf.extend_from_slice(&7u32.to_le_bytes());
assert!(matches!(
DataLayout::parse(&buf, 8, 8).unwrap_err(),
FormatError::UnexpectedEof { .. }
));
// Compact, raw data shorter than its declared size.
let mut buf = v1v2_header(2, 1, 0);
buf.extend_from_slice(&4u32.to_le_bytes());
buf.extend_from_slice(&100u32.to_le_bytes());
buf.extend_from_slice(&[0; 4]);
assert!(matches!(
DataLayout::parse(&buf, 8, 8).unwrap_err(),
FormatError::UnexpectedEof { .. }
));
}
#[test]
fn v3_compact() {
let mut buf = vec![3u8, 0]; // version=3, class=0 (compact)
@@ -602,6 +971,7 @@ mod tests {
chunk_index_type: None,
single_chunk_filtered_size: None,
single_chunk_filter_mask: None,
dont_filter_partial_edge_chunks: false,
}
);
}
@@ -679,10 +1049,35 @@ mod tests {
chunk_index_type: Some(1),
single_chunk_filtered_size: None,
single_chunk_filter_mask: None,
dont_filter_partial_edge_chunks: false,
}
);
}
#[test]
fn v4_chunked_dont_filter_partial_edge_chunks_flag() {
let mut buf = vec![4u8, 2]; // version=4, class=2
buf.push(0x01); // flags bit 0 = don't filter partial edge chunks
buf.push(2); // dimensionality=2
buf.push(4); // dim_size_encoded_length=4
buf.extend_from_slice(&5u32.to_le_bytes());
buf.extend_from_slice(&4u32.to_le_bytes());
buf.push(3); // Fixed Array
buf.push(10); // max_dblk_page_nelmts_bits
buf.extend_from_slice(&0x3000u64.to_le_bytes());
match DataLayout::parse(&buf, 8, 8).unwrap() {
DataLayout::Chunked {
dont_filter_partial_edge_chunks,
btree_address,
..
} => {
assert!(dont_filter_partial_edge_chunks);
assert_eq!(btree_address, Some(0x3000));
}
other => panic!("expected Chunked, got {other:?}"),
}
}
#[test]
fn v4_chunked_single_chunk_with_filters() {
let mut buf = vec![4u8, 2]; // version=4, class=2
@@ -705,6 +1100,7 @@ mod tests {
chunk_index_type: Some(1),
single_chunk_filtered_size: Some(1024),
single_chunk_filter_mask: Some(0),
dont_filter_partial_edge_chunks: false,
}
);
}
@@ -815,6 +1211,62 @@ mod tests {
assert_eq!(v1.iter_linear_1d(8).unwrap(), vec![4, 5, 6, 7]);
}
#[test]
fn parse_vds_mappings_v1_shared_names() {
// Written by HDF5 2.0 (h5py, libver=("v200", "v200")) for three
// mappings from `a_rather_long_source_file.h5:a_rather_long_dataset_name`
// and one from the same file: the entries carry flags 0x00, 0x03, 0x03
// and 0x06, so names after the first are stored as entry indices.
let blob: &[u8] = &[
0x01, 0x04, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x61, 0x5f, 0x72, 0x61,
0x74, 0x68, 0x65, 0x72, 0x5f, 0x6c, 0x6f, 0x6e, 0x67, 0x5f, 0x73, 0x6f, 0x75, 0x72,
0x63, 0x65, 0x5f, 0x66, 0x69, 0x6c, 0x65, 0x2e, 0x68, 0x35, 0x00, 0x61, 0x5f, 0x72,
0x61, 0x74, 0x68, 0x65, 0x72, 0x5f, 0x6c, 0x6f, 0x6e, 0x67, 0x5f, 0x64, 0x61, 0x74,
0x61, 0x73, 0x65, 0x74, 0x5f, 0x6e, 0x61, 0x6d, 0x65, 0x00, 0x02, 0x00, 0x00, 0x00,
0x03, 0x00, 0x00, 0x00, 0x01, 0x02, 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, 0x00,
0x01, 0x00, 0x04, 0x00, 0x02, 0x00, 0x00, 0x00, 0x03, 0x00, 0x00, 0x00, 0x01, 0x02,
0x02, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, 0x00, 0x01, 0x00, 0x01, 0x00, 0x00, 0x00,
0x01, 0x00, 0x01, 0x00, 0x04, 0x00, 0x03, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0x00, 0x00, 0x00, 0x03,
0x00, 0x00, 0x00, 0x01, 0x02, 0x01, 0x00, 0x00, 0x00, 0x04, 0x00, 0x01, 0x00, 0x01,
0x00, 0x04, 0x00, 0x02, 0x00, 0x00, 0x00, 0x03, 0x00, 0x00, 0x00, 0x01, 0x02, 0x02,
0x00, 0x00, 0x00, 0x01, 0x00, 0x01, 0x00, 0x01, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01,
0x00, 0x01, 0x00, 0x04, 0x00, 0x03, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0x00, 0x00, 0x00, 0x03, 0x00,
0x00, 0x00, 0x01, 0x02, 0x01, 0x00, 0x00, 0x00, 0x08, 0x00, 0x01, 0x00, 0x01, 0x00,
0x04, 0x00, 0x02, 0x00, 0x00, 0x00, 0x03, 0x00, 0x00, 0x00, 0x01, 0x02, 0x02, 0x00,
0x00, 0x00, 0x02, 0x00, 0x01, 0x00, 0x01, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01, 0x00,
0x01, 0x00, 0x04, 0x00, 0x06, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02,
0x00, 0x00, 0x00, 0x03, 0x00, 0x00, 0x00, 0x01, 0x02, 0x01, 0x00, 0x00, 0x00, 0x00,
0x00, 0x01, 0x00, 0x01, 0x00, 0x04, 0x00, 0x02, 0x00, 0x00, 0x00, 0x03, 0x00, 0x00,
0x00, 0x01, 0x02, 0x02, 0x00, 0x00, 0x00, 0x03, 0x00, 0x01, 0x00, 0x01, 0x00, 0x01,
0x00, 0x00, 0x00, 0x01, 0x00, 0x01, 0x00, 0x04, 0x00, 0x8e, 0xa7, 0xea, 0x7a,
];
let mappings = parse_vds_mappings(blob, 8).unwrap();
let names: Vec<(&str, &str)> = mappings
.iter()
.map(|m| (m.source_file.as_str(), m.source_dataset.as_str()))
.collect();
let (file, dset) = ("a_rather_long_source_file.h5", "a_rather_long_dataset_name");
assert_eq!(
names,
vec![(file, dset), (file, dset), (file, dset), (".", dset)]
);
}
#[test]
fn parse_vds_mappings_v1_forward_reference_is_error() {
// Entry 0 claiming to share entry 0's file name must not index past
// the entries decoded so far.
let mut blob = vec![0x01u8, 1, 0, 0, 0, 0, 0, 0, 0, 0x01];
blob.extend_from_slice(&[0u8; 8]);
blob.extend_from_slice(b"d\0");
assert!(parse_vds_mappings(&blob, 8).is_err());
// Unknown flag bits are refused.
let blob = [0x01u8, 1, 0, 0, 0, 0, 0, 0, 0, 0x08, b'd', 0];
assert!(parse_vds_mappings(&blob, 8).is_err());
}
#[test]
fn parse_vds_mappings_external_v0() {
// Block version 0 with an explicit (external) source file name.
@@ -862,4 +1314,44 @@ mod tests {
let blob = [0x01u8, 0, 0, 0, 0, 0, 0, 0, 0];
assert!(parse_vds_mappings(&blob, 8).unwrap().is_empty());
}
/// A virtual dataset's mappings resolve identically through a
/// read_at-only CountingStorage, in two reads of the global heap.
#[test]
fn vds_mappings_through_storage_match_slice() {
use crate::message_type::MessageType;
use crate::object_header::ObjectHeader;
use crate::storage::CountingStorage;
let file: &[u8] = include_bytes!("../tests/fixtures/vds_same_file.h5");
let sb = crate::superblock::Superblock::parse(file, 0).unwrap();
let (os, ls) = (sb.offset_size, sb.length_size);
let storage = CountingStorage::new(file.to_vec());
let mut virtuals = 0;
for child in
crate::group_v2::resolve_group_children(file, &sb, sb.root_group_address).unwrap()
{
let h =
ObjectHeader::parse(file, child.object_header_address as usize, os, ls).unwrap();
let Some(msg) = h
.messages
.iter()
.find(|m| m.msg_type == MessageType::DataLayout)
else {
continue;
};
let mut want = DataLayout::parse(&msg.data, os, ls).unwrap();
if !matches!(want, DataLayout::Virtual { .. }) {
continue;
}
let mut got = want.clone();
want.resolve_vds_mappings(file, ls).unwrap();
storage.reset();
got.resolve_vds_mappings_in(&storage, ls).unwrap();
assert_eq!(format!("{got:?}"), format!("{want:?}"));
assert!(matches!(&got, DataLayout::Virtual { mappings, .. } if !mappings.is_empty()));
assert_eq!(storage.reads(), 2);
virtuals += 1;
}
assert!(virtuals >= 1);
}
}
crates/clawhdf5-format/src/data_read.rs
+1181 -517
View File
File diff suppressed because it is too large Load Diff
crates/clawhdf5-format/src/dataspace.rs
+67 -13
View File
@@ -7,6 +7,9 @@ use alloc::vec::Vec;
use crate::error::FormatError;
/// Most dimensions a dataspace can have (`H5S_MAX_RANK`).
pub const MAX_RANK: u8 = 32;
/// Type of dataspace.
#[derive(Debug, Clone, PartialEq)]
pub enum DataspaceType {
@@ -67,6 +70,12 @@ impl Dataspace {
let version = data[0];
let rank = data[1];
let flags = data[2];
// H5O__sdspace_decode's checks.
if rank > MAX_RANK {
return Err(FormatError::InvalidDataspace(
"simple dataspace dimensionality is too large",
));
}
let (space_type, header_size) = match version {
1 => {
@@ -88,6 +97,11 @@ impl Dataspace {
2 => DataspaceType::Null,
_ => return Err(FormatError::InvalidDataspaceType(type_byte)),
};
if st != DataspaceType::Simple && rank > 0 {
return Err(FormatError::InvalidDataspace(
"invalid rank for scalar or NULL dataspace",
));
}
(st, 4usize)
}
_ => return Err(FormatError::InvalidDataspaceVersion(version)),
@@ -107,8 +121,13 @@ impl Dataspace {
// Read max dimensions if flags bit 0 is set
let max_dimensions = if flags & 0x01 != 0 {
let mut max_dims = Vec::with_capacity(rank as usize);
for _ in 0..rank {
for &dim in &dimensions {
let val = read_length(data, pos, length_size)?;
if dim > val {
return Err(FormatError::InvalidDataspace(
"dataspace dimension size is greater than its maximum size",
));
}
max_dims.push(val);
pos += ls;
}
@@ -176,7 +195,6 @@ impl Dataspace {
match self.space_type {
DataspaceType::Null => Ok(0),
DataspaceType::Scalar => Ok(1),
DataspaceType::Simple if self.dimensions.is_empty() => Ok(0),
DataspaceType::Simple => self
.dimensions
.iter()
@@ -195,18 +213,14 @@ impl Dataspace {
match self.space_type {
DataspaceType::Null => 0,
DataspaceType::Scalar => 1,
DataspaceType::Simple => {
if self.dimensions.is_empty() {
0
} else {
// Saturate rather than wrap: a wrapped product could
// under-size a buffer. Size-critical callers use
// `checked_num_elements`.
self.dimensions
// A simple dataspace of rank 0 holds one element, as in libhdf5
// (the product of no dimensions). Saturate rather than wrap: a
// wrapped product could under-size a buffer. Size-critical
// callers use `checked_num_elements`.
DataspaceType::Simple => self
.dimensions
.iter()
.fold(1u64, |acc, &d| acc.saturating_mul(d))
}
}
.fold(1u64, |acc, &d| acc.saturating_mul(d)),
}
}
}
@@ -352,4 +366,44 @@ mod tests {
let ds = Dataspace::parse(&data, 8).unwrap();
assert_eq!(ds.max_dimensions, Some(vec![10]));
}
/// A simple dataspace of rank 0 (cve-2020-18494's `/dset1`) holds one
/// element in libhdf5, which h5py reads as shape `()`. It was 0.
#[test]
fn simple_rank_zero_holds_one_element() {
let data = build_v2_dataspace(0, 0, 1, &[], None);
let ds = Dataspace::parse(&data, 8).unwrap();
assert_eq!(ds.space_type, DataspaceType::Simple);
assert_eq!(ds.num_elements(), 1);
assert_eq!(ds.checked_num_elements().unwrap(), 1);
}
/// `H5O__sdspace_decode`'s checks.
#[test]
fn refuses_what_libhdf5_refuses() {
let too_many = build_v2_dataspace(33, 0, 1, &[1; 33], None);
assert!(matches!(
Dataspace::parse(&too_many, 8),
Err(FormatError::InvalidDataspace(_))
));
let scalar_with_rank = build_v2_dataspace(1, 0, 0, &[4], None);
assert!(matches!(
Dataspace::parse(&scalar_with_rank, 8),
Err(FormatError::InvalidDataspace(_))
));
let null_with_rank = build_v2_dataspace(1, 0, 2, &[4], None);
assert!(matches!(
Dataspace::parse(&null_with_rank, 8),
Err(FormatError::InvalidDataspace(_))
));
let over_max = build_v1_dataspace(2, 0x01, &[5, 20], Some(&[10, 10]));
assert!(matches!(
Dataspace::parse(&over_max, 8),
Err(FormatError::InvalidDataspace(_))
));
// 32 dimensions, and a size equal to the maximum or unlimited, are fine.
assert!(Dataspace::parse(&build_v2_dataspace(32, 0, 1, &[1; 32], None), 8).is_ok());
let at_max = build_v1_dataspace(2, 0x01, &[10, 20], Some(&[10, u64::MAX]));
assert!(Dataspace::parse(&at_max, 8).is_ok());
}
}
crates/clawhdf5-format/src/datatype.rs
+965 -20
View File
File diff suppressed because it is too large Load Diff

Some files were not shown because too many files have changed in this diff Show More