Since the M2 merge the facade handed in-memory files to the generic
`*_in` parsers as `&[u8]` (`with_bytes!`), which instantiates them in
the facade crate, where the format crate's private helpers do not
inline without LTO: listing a 400-group v1 file through `File::open`
was 7-10% slower than main. `ObjectHeader::parse_in`,
`group_v2::{resolve_child_in, resolve_group_children_in,
resolve_path_any_in}` and `attribute::{extract_attributes_tolerant_in,
find_attribute_in}` now pass a storage with `as_contiguous()` to their
non-generic slice entry point, compiled once in the format crate; other
storages reach the same generic core as before.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Reading continuation chunks from a queue (7e5e920, a69c5be) allocated a
queue Vec and a BTreeSet of chunk starts for every header, and inlined
the per-chunk message loop into the generic parser: ObjectHeader::parse
over 401 version-1 headers went from 24.8 to 45.7 us.
ChunkSpans now keeps the first 8 chunks in an inline array (cycle check
by scan) and is also the read queue; only a header of more chunks
allocates (a boxed spill list and start set). The message loop of one
version-1 chunk is its own non-generic function. Same checks as before:
any number of chunks up to 65,536, cycles refused, chunks bounded by the
file size, one chunk buffer alive at a time, libhdf5 message order,
overlap allowed. The cycle test now also covers spilled chunk lists.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The M2 fetch bound gave every codec n + n/4 + 4096 stored bytes, but
ZFP's fixed-rate mode stores up to 64 bits per value, doubling 4-byte
types: 60 of the 2205 zfp_interop datasets (rate 64, f4/i4) failed with
'compressed stream ends early' once M2 and ZFP were merged.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
check-32bit-casts.sh flagged two casts added by the previous commits; both
values are bounded (checked by gather_storage's first walk, and built from
a usize fetch length), so they go through addr::saturating_usize.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The facade equivalence harness turned every data-read error into "Err", so
it could not see File::open_storage failing differently from File::open
(a Storage or ContiguousStorageRequired error where the mmap path gives a
decode error, say).
- value() keeps the whole error. The only allowance is for a line on which
File::open itself varies between opens — the chunk cache lists a damaged
dataset's chunks in hash-map order, so which failing chunk a full read
reports varies (cve-2025-2310.h5, the one corpus file where this shows):
both sides must fail there, and a fresh File::open (up to 64) must
reproduce the storage's exact error. Open errors were already compared
in full; they still agree.
- The storage transcript may not contain ContiguousStorageRequired.
- More selections: a strided hyperslab (every third row) through
read_f64_selection, and out-of-order points through read_selection and
read_i64_selection.
- harness_compares_errors_not_just_failures checks the harness itself:
two different errors are different values, and a difference File::open
does not produce is reported.
With full errors the harness passes on the 61 fixtures and on the corpus
(701 files, 621 open).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
An attribute needing a heap block larger than the next one was refused
("skipping blocks too small for an object", "a first object too large for
the starting block"); once an object's move to dense storage was refused
it refused every new attribute, so 24% of set_attr calls in the review's
random workload failed.
Following H5HF__hdr_update_iter, H5HF__man_iblock_root_create/_double and
H5HF__hdr_skip_blocks, the smaller blocks are now skipped: the iterator
moves past them and they become an indirect free section with a first
row section (serialized, class 1, as H5HF__sect_indirect_serialize writes
it) and ghost normal rows, added as returned space so it merges with a
range skipped just before it (H5HF__sect_indirect_merge_row). Later
objects that best-fit a row section get a block created there
(H5HF__man_iblock_alloc_row / H5HF__sect_indirect_reduce_row: from the
start or end of the range, or from its middle, which splits it, with
libhdf5's span bookkeeping). Heaps with such sections, as libhdf5 writes
them, are now read too (they were refused at open).
dense_skipped_blocks_match_libhdf5 drives every path (merge, split, end,
last entry, row wrap) on earliest/v110/latest files against libhdf5
doing the same edits one session each; heaps, free sections and index
B-trees are equal after every phase. The refusal test now checks the
skip against libhdf5 and keeps a real refusal (last object in a block);
clawhdf5-written heaps get 1-4 KiB attributes too. The three tests fail
on the previous fheap.rs. Random workload refusals: 24% -> 2.2%, all the
documented last-object-in-a-block case.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
306 104 and 49 930 are h5repack of the editor's file; the text read as if
they were h5repack of libhdf5's, which measures 305 954 and 50 188. Both
are now given, from measure_append_waste rerun on 2026-09-26 (file sizes
unchanged).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The version-1 chunk walk nested continuation chunks depth-first and kept
every enclosing chunk's buffer alive, up to 65 536 chunks. With storage
that hands out owned buffers (CountingStorage, the Storage trait, remote
storage) a crafted chain of chunks nested in each other read and held the
square of the file's size (a 192 KB file read 768 MB).
Chunks are now read from a FIFO queue of (address, length) pairs in the
order their continuation messages are found, as H5O_protect does and as
the editor's header walker already did, each buffer released before the
next read. In both header versions a chunk starting at an address seen
before (cycle) is refused, and so are chunks adding up to more than the
file, which bounds a header's reads by the file's size. Overlap itself is
allowed: libhdf5 reads cve-2025-7067.h5, whose continuation chunk overlaps
chunk 0 (refusing overlap cost that conformance file).
Tests: the nested chain is refused having read at most the file (it read
n^2 bytes before); a 3000-chunk chain reads each chunk once; a chunk's
messages follow the whole previous chunk (they were inserted at the
continuation message); an overlapping continuation chunk is read.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
prune_plan stored one Vec<u64> for every chunk coordinate of the region a
shrink cuts off, existing or not, so a sparse dataset exhausted memory
(about 62 bytes per coordinate; (4, 2e7) with chunks (1, 1) took 2.5 GB,
larger extents never finished). It now places each existing chunk in
H5D__chunk_prune_by_extent's walk (its pass, then its coordinates) and
sorts, which gives the same chunks, order and actions in memory and time
proportional to the chunks that exist.
A unit test checks the plan against the full walk (kept as the test's
reference) for 3000 random extents and chunk subsets. The interop test
shrinks a (4, 10^12) dataset with chunks (1, 1) and 9 chunks (v1 and v2
B-tree): 0.56 s and 43 MB peak; the old code aborted on allocation under an
8 GB limit.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Our checksum reduced its sums with `% 65535`; libhdf5's
H5_checksum_fletcher32 folds them with `(s & 0xffff) + (s >> 16)`, which
leaves 0xffff where the modulo leaves 0. On about one chunk in 32768
libhdf5 refused the chunks we wrote and we refused the chunks it wrote.
Every release since v2.1.0 is affected.
clawhdf5_format::checksum::fletcher32 is a port of H5_checksum_fletcher32
and the filter's only implementation. Verification also accepts the
byte-swapped form libhdf5 accepts (1.6.2 and earlier) and the `% 65535`
form earlier releases wrote, so their files stay readable.
The new interop test compares the checksum with libhdf5's own function
(ctypes) on every 1- and 2-byte input and 40 000 random and fold-heavy
inputs, and moves fold-case chunks between h5py and FileBuilder/FileEditor
in both directions; with the old filters.rs the three file tests fail.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
ExtentBytes and read_exact_at/read_upto rejected short results but passed
longer-than-asked ones through, and FileData forwarded them too, so a
Storage that broke read_at's contract by returning extra bytes had them
decoded or returned as data (a contiguous dataset read gained 37 junk
bytes). gather_storage alone trimmed.
- storage::exact_len (new, pub): a read of len bytes as exactly len — cut
when longer, an error when short. read_exact_at, read_upto and
ExtentBytes (so chunk fetches and selection gathers) go through it.
- FileData cuts a backend's answer to what it asked for before laying the
cache image over it.
- Tests: over a storage that appends 37 junk bytes to every read, every
format-crate fixture reads exactly as from the slice
(overlong_reads_are_cut_to_the_range_asked_for), and every facade
fixture opens and reads through File::open_storage as through File::open
(overlong_storage_reads_identically). Both failed before.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Since M2 the facade handed the metadata parsers its FileData view, so a
local file ran the parsers monomorphised for FileData, whose Storage impl
worked out contiguous() (patched and overlay checks, two range conversions)
on every structure read. A metadata walk of h5stat_newgrat.h5 (35,001
groups: open, entries and attrs of each) was about 4.5% slower than at
8f59b2e.
- FileData works out its contiguous slice once at open (a borrow of its own
heap/mapped buffer, kept as a pointer; see the SAFETY notes).
- with_bytes! hands the in-memory slice to the format parsers when the file
has one (header parsing, attributes, group listings and lookups, path
resolution, shared messages, VL decoding), so local files run the [u8]
parsers as before; storage-backed files still get FileData.
Provisional A/B on tank (load 5-11), best of 30, 5 alternating rounds:
walk 26.07-26.37 ms at 8f59b2e, 27.24-28.03 ms before this commit,
26.52-26.98 ms after. Caching alone did not move it (27.02-27.47 ms); the
dispatch did. File::open read_f32 on 32M f32 (contiguous, chunked, gzip)
stays within noise of 8f59b2e.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
checked_div in the test server's throttle, a slice for the single range
of fetch_first, and dead-code allowances for the redaction helpers in a
build with neither http nor a cloud store.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
CHANGELOG, the clawhdf5-remote and h5rs READMEs and the remote-files
known issues: redirect rules, scaled timeouts (min_speed), URL redaction,
claimed lengths never allocated (download, --max-download), a 200 for a
small file accepted, and ObjectStoreStorage from any thread.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
open_arg_whole opened and parsed the remote file through open_arg, then
opened it again to download it, so every `h5rs check URL` probed the
server twice. It now opens the storage once and downloads through the
same block cache (whose first block the probe already filled).
Test: check --data of a file within one block costs exactly one request
(two before).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
cached() mapped an error of its open-time prefetch (a network error, a
changed file) to Error::Hdf5(Format(Storage)), misclassifying it for
callers that match on the variant. It is now Error::Remote
(RemoteError::Backend with the backend's message). open_object and the
s3/gs/az URLs fetch the first block of an ObjectStoreStorage directly, so
their errors keep their kind (FileChanged, ObjectStore).
Test: cached() over a backend whose reads fail gives Error::Remote(Backend)
(Error::Hdf5 before).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
It refused whenever Handle::try_current() was Ok, which is also the case
inside spawn_blocking threads — so the workaround its own error message
recommended failed the same way, and the backend could only be used from
a bare std::thread in a tokio application.
Reads are now spawned on the storage's own runtime and the caller waits on
a channel: the future never runs on the caller's thread, so neither a
spawn_blocking thread nor a current-thread runtime can deadlock or panic
(a read inside a runtime blocks that thread, like any blocking call; the
docs still recommend spawn_blocking there).
Tests: a read in spawn_blocking of a multi-thread runtime and a read inside
a current-thread runtime's task give File::open's values (both errors
before).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
gather_storage merged only runs that touch, so a strided selection of a
contiguous dataset over a Storage became one range (and one owned Vec) per
element: a stride-2 read of 32M f32 through File::open_storage made
16,777,232 read_at calls, took 2.0 s and peaked at 2.09 GB.
The selection is now walked twice. The first walk checks the runs and plans
spans: runs in increasing order at most 4 KiB apart (GATHER_GAP_BYTES) are
read as one span up to 8 MiB (GATHER_SPAN_BYTES; a longer run is split), so
nothing is stored per run. The spans are fetched in RAW_BATCH_BYTES batches
while the second walk copies each run out of its span. Same checks and
errors as before.
The same read is now 32 reads and 0.31 s (File::open: 0.08 s).
contiguous_read_interop: every h5py-checked selection is also read through
File::open_storage and must give libhdf5's bytes; a new test bounds the
range reads of strided, blocked, column and point selections (stride 2: at
most 1 data read; 563,200 before).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
timeout_global (60 s) covered a whole request, and a request can carry
8 MiB (max_request): below about 140 KB/s every block run timed out, was
retried from scratch and failed, so a slow link could not read remote
files at all.
HttpOptions::timeout (now 30 s) bounds connecting and receiving the
response headers; the body gets timeout + its size at the new
HttpOptions::min_speed (16 KiB/s by default: 94 s for a 1 MiB block).
A slow but moving link is not cut off; a stalled one still fails.
(ureq has no idle timeout; its body timeout is a total budget.)
The test server can throttle bodies and stall mid-body. Test: a 256 KiB
block at 256 KiB/s reads with a 300 ms timeout (it failed before), and a
body stalled for 20 s fails in under 5 s.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The first request asks for bytes=0-1048575. RFC 9110 lets a server answer
200 when the range covers the whole representation, so a file under
1 MiB on a server that does support ranges could be refused as 'does not
support range requests'. A 200 whose Content-Length (or, without one, its
body, read at most that far) is within the range asked for is now kept as
the whole file and read from memory; a longer one is still refused unless
allow_full_download is set.
Test: a 9968-byte file served with 200 opens in one request with the
transcript of File::open (it was refused before); with a 4096-byte first
request it is still refused.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
ureq's defaults followed up to 10 redirects, including from https to plain
http, and forwarded the custom HttpOptions::headers (X-Api-Key, Cookie,
...) to whatever host a redirect named — only Authorization was stripped.
HttpStorage now follows redirects itself (ureq's max_redirects is 0):
- at most HttpOptions::max_redirects per request (default 5; 0 refuses
any redirect), then RemoteError::Redirect;
- never from https to another scheme, nor to a non-http(s) URL;
- once a redirect leaves the URL's origin (scheme, host, port), none of
the custom headers is sent any more (Authorization included);
- each hop counts as a request; errors show the target redacted.
Tests: a redirect to another local port reads the right data and the
target never sees X-Api-Key or Authorization (it did before); a
same-origin redirect keeps them; a loop stops after 6 requests; 0 refuses;
unit tests for target resolution, the https downgrade and origins.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Only the full read split its chunk fetches into 64 MiB batches. The
selection path, the indexed read and the parallel_read decoders fetched
every chunk's stored bytes in one read_ranges call, each extent bounded only
by the file length, so a crafted chunk index pointing many chunks at one
large extent made File::open_storage hold chunks x extent bytes (3.3 GB from
a 16.8 MB file) before the first decode error.
- storage::for_each_extent_batch is now the one way raw-data reads fetch
chunk bytes: batches of at most RAW_BATCH_BYTES (now pub), each decoded
before the next is fetched. Used by the full, cached, indexed, selection
and parallel_read paths; the sweep read uses read_extent per chunk.
- ExtentReq carries each chunk's claimed extent (bounds-checked as before,
same errors) and the prefix actually fetched:
filters::stored_chunk_limit — the chunk size if unfiltered, else each
applied filter's worst-case growth (n + n/4 + 4096 per codec; unbounded
only for an application-registered codec). The in-memory path cuts the
slice it decodes the same way, so both paths still agree.
- tests/raw_fetch_bounds.rs: a crafted chunked_large.h5 (ten chunks all
claiming 20 MiB at one padding blob) read through every path over a
storage that records the largest single fetch; and 160 MiB of legitimate
unfiltered chunks fetched batch by batch. Before: one 80 MiB fetch
(selection) and one 160 MiB fetch; after: within the budget.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Every RemoteError message and HttpStorage's Debug output held the URL as
given, with any user:password@ and the query string — for a presigned
S3/GCS/Azure URL, its signature or token. An application logging the
error leaked the credential.
- New clawhdf5_remote::redact_url: no userinfo, no fragment, query values
replaced by REDACTED (plain key names kept).
- HttpStorage formats every message with the redacted URL, and scrubs the
URL's secret parts from errors of the HTTP client (whose texts can echo
the URI); Debug shows the redacted URL. storage_for_url's and the object
store URL errors are redacted too. HttpStorage::url() still returns the
URL as given, documented as not for logging.
- h5rs prints FILE arguments that are URLs redacted: in errors and in
dump/stat/check/diff output.
- The test server can force a status and send a wrong Content-Range.
Tests: 404, 403 (at open and on a read), wrong Content-Range (at open and
on a read), no range support, encoded body, ETag change, timeout,
connection closed and bad scheme errors, Display and Debug, contain none
of the secrets; h5rs likewise for every subcommand.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
h5rs check URL read the whole file with one read_at(0, len), len being
whatever Content-Range said. BlockCache listed every block index of the
span and preallocated len bytes: a server claiming 2^62 bytes for a 10 KB
file made h5rs abort (memory allocation of 35184372088832 bytes failed).
- BlockCache: a read spanning more than the budget (or eight max_requests)
is fetched piece by piece and not kept, its output growing only as
data arrives; read_ranges falls back to that per range; prefetch is
clamped to the budget.
- New clawhdf5_remote::download(storage, max_bytes): refuses a claimed
length above the limit (RemoteError::TooLarge) before any request, then
reads in 64 MiB steps. New RemoteError::Backend for read errors.
- h5rs check downloads through it, with --max-download N (default 1 GiB).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
A server can claim any length in Content-Range. block_len computed
start + block_size, which overflowed in the last blocks of a file claimed
to be near u64::MAX (a panic in debug builds, a wrapped value in
release); insert() multiplied block indices unchecked. The cache's block
arithmetic is now saturating/checked, and a run that does not split into
whole blocks is an error instead of an endless loop or a slice panic.
The test server gains fake_total (claim a length, serve zeros past the
data); a test reads the last bytes of such files and opens a file whose
superblock EOF and root addresses sit near u64::MAX.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
A local port scanner's GET / reached the test listeners and was counted,
failing the exact request budgets (and consuming injected 503s). Requests
for paths the server does not serve are now answered 404 without being
counted, delayed or failed; the query string is not part of the path.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
A reader that waited for another reader's fetch of a block got "the
fetch of this block failed" when that fetch failed, not why: a file
replaced on the server while open was reported as FileChanged to one
thread and as an anonymous failure to the others. The fetch's error is
now handed to every reader waiting on it.
Regression test: four threads read the same block from a slow backend
whose fetches fail with a "changed while open" error; each gets that
error (it failed for the waiters before this change).
Also fixes the ignore-Range test, broken by the previous commit: the
test server now counts a body before sending it, so "the refused body
was not read" is checked as "refused at the first response".
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
README: "Reading remote files" (open_url, the range_server and read_url
examples with their real output against the fixtures, h5rs on URLs), the
crate in the crate map and the unreleased highlights. CHANGELOG: the
clawhdf5-remote crate, h5rs URLs, File::storage and
VlResolver::element_in, with the request counts over the conformance
corpus (tank, 2026-09-26, the command given). known-issues: the M2
range-read entry updated (the cache now exists; h5rs reads through
storage) and a new entry for the remote backends' limits (no Python or
browser URLs yet, fixed block size, cloud stores not run against a real
bucket, validators, credentials). Design doc: M3 status with the choices
that differ from the plan (a crate rather than a clawhdf5-io feature,
ureq for HTTP so the default build has no C) and the corpus counts.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The per-file report now separates a tree view (open, every group's
entries, every dataset's shape and type) from reading the largest
dataset under 64 MiB, and checks the budget the design's testing section
asks for: listing the IMERG file (file A of docs/design/range-reads.md
section 2) takes at most 3 requests when CLAWHDF5_REMOTE_CORPUS includes
it. The test server now counts a response's bytes before sending it: a
client could read a body and reset the counters before the server thread
had added it, so the counts of the next file were occasionally too high.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Datasets with early allocation and unlimited dimensions (Extensible
Array, version-2 B-tree, version-1 B-tree under earliest), unfiltered
and deflated: the random resize workload gives the values h5py gets and
the same chunk index shape, with every chunk a growth brings in
allocated and filled as H5D__chunk_allocate does.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
With the `remote` feature (`remote-https` for https://), ls, dump, stat
and diff take an http(s):// (or s3://, gs://, az:// with those
clawhdf5-remote features) URL wherever they take a file, and read it by
range requests through clawhdf5-remote's block cache. check validates
every byte, so it downloads a remote file whole and checks it as before.
Without the feature a URL is a clean error naming it.
The tools read the file through File::storage instead of as_bytes: object
headers, shared messages, attributes, v1 and v2 group links, dense
storage (fractal heaps and v2 B-trees), path resolution, chunk listings
and variable-length values go through the format crate's *_in functions,
and the fractal-heap block verifier reads each block through the storage
(a read failure of a remote file is reported as a problem, not as "past
the end of the file"). A local file's storage is its mapped bytes, so its
reads are still slices. stat's file size comes from the opened file, so
it is right for a URL.
Tests: tests/remote.rs serves fixtures (old and new formats, a paged
file, a metadata cache image, a multi-block fractal heap, compounds, v1
groups) with the clawhdf5-remote test server and requires every
subcommand's output and exit status for the URL to equal the local
file's, and diff of the two to be clean; 404s, non-HDF5 bodies and
https without its feature are clean errors. Local output is unchanged:
the old and new h5rs print the same for ls -r -v, dump, stat and check
--data on the 747 conformance and CVE corpus files (tank, 2026-09-26;
the dumps of h5diff_hyper1/2.h5 were too large for the comparison
script, their ls, stat and check agree), except cve-2025-2310.h5, whose
dump error messages differ between runs of the old binary too (which
failing chunk is reported first).
ci-test.sh lints h5rs with remote-https, runs the URL tests and checks
h5rs with remote for C.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
VlResolver::element and string_element return slices of the whole file,
so they exist only for a resolver over &[u8]. Their *_in forms work for
any Storage (a remote file): the element's bytes borrowed from the
resolver's cache of heap collections, with the same null-element, NUL
and size checks. h5rs decodes variable-length values with them.
Test: over a read_at-only storage they give what element/string_element
give over the slice, for a string with an embedded NUL, a null element
and an element whose heap object has the wrong size.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The bytes File::as_bytes returns (from the superblock on, bounded by the
recorded end of file, a metadata cache image laid over), as a
&(dyn Storage + Send + Sync) for every backend. Code that parses the file
itself with the clawhdf5_format *_in functions — h5rs does — can then
read a file opened with File::open_storage (a remote file) as well as a
local one; in memory its as_contiguous() is as_bytes(), so local reads
stay slices.
Test: for every fixture, File::open's storage() is as_bytes() as its
contiguous view, and File::open_storage over a read_at-only storage gives
the same bytes through storage().read_at.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
ObjectStoreStorage (feature `object-store`, pure Rust) reads one object
of any object_store store by ranged get_opts, pinned at open by a head
request: If-Match with its ETag (and the ETag and size of every response
compared), else its version or modification time. A change is
RemoteError::FileChanged. object_store is async and Storage is not, so
the storage owns a small multi-threaded tokio runtime (two workers) and
blocks the calling thread on it; the ranges of one read_ranges call are
fetched concurrently (up to 8). From inside another tokio runtime it
refuses with RemoteError::Usage instead of blocking a worker, and it
shuts its runtime down in the background on drop so dropping it in async
code does not panic.
open_object(store, path, options) opens a file through a block cache
(first block prefetched); open_url accepts s3://, gs:// and az:// with
the `s3`, `gcs` and `azure` features, configured from the environment by
object_store's from_env builders. Those pull object_store's cloud clients
and aws-lc-rs (C), so they are opt-in; without them the URL is a clean
UnsupportedScheme error naming the feature.
Tests against object_store's in-memory and local-file stores (no cloud):
every fixture's transcript equals File::open's, a multi-block object is
fetched in coalesced block runs, an object replaced while open is an
error, and a missing object or a read from inside a runtime is a clean
error. ci-test.sh lints all backends, runs these tests (with s3 for its
URL parsing test) and checks object-store for C in the no-C step.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
CHANGELOG (Unreleased): the new FileEditor operations, space reuse, and
the two reader fixes (implicit index grid, object-header continuation
chains). known-issues: the editor's remaining refusals (skipped heap
blocks, heaps with filters or child indirect blocks, freeing a heap
block, implicit-index insertions, ...) and the append-waste sizes before
and after reuse (measure_append_waste, tank 2026-09-26; file sizes are
deterministic). range-reads design: status note on the reader changes.
README and CLAUDE.md: what the editor covers and how to test it.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Range-read milestone M3, first half: a new crate with the block cache the
design makes mandatory for remote files and an HTTP backend, so
open_url("http://...") gives a clawhdf5::File over File::open_storage.
BlockCache wraps any Storage: aligned blocks (1 MiB by default, the size
docs/design/range-reads.md section 2 measured), LRU with a byte budget,
the missing blocks of one read_at/read_ranges fetched with one backend
read_ranges call as runs of consecutive blocks (a one-block gap filled to
merge runs, each request at most 8 MiB), and reads that miss more than
half the budget not kept. Thread-safe without holding the lock across a
fetch: a block being fetched is in flight, a second reader waits for it
instead of fetching it again, and a failed fetch fails its waiters and is
not cached. A backend holding the file in memory passes through.
HttpStorage (ureq, no TLS by default; `https` adds rustls with ring):
opening is one ranged GET of the first block, whose Content-Range gives
the length (the cache keeps the bytes). The file is pinned by a strong
ETag (If-Match), else Last-Modified (If-Unmodified-Since), and its length,
checked on every response: a change is RemoteError::FileChanged, never
mixed data. A server that ignores Range is refused without reading the
body unless a full download is allowed. Connection errors, timeouts,
408/429/5xx and short bodies are retried with exponential backoff;
Accept-Encoding: identity, and an encoded body is refused. read_ranges
fetches its ranges in parallel.
Tests (a std-only HTTP/1.1 server in tests/common/server.rs, also the
range_server example): every fixture read over HTTP gives File::open's
transcript (CLAWHDF5_REMOTE_CORPUS adds the conformance corpus), with
request counts per file with and without the cache; an h5py-written file
against libhdf5's values; a multi-block file fetched in whole blocks, each
once; a server ignoring Range; a file replaced mid-read (ETag,
Last-Modified, length only); truncated bodies and 503s (retried, then an
error, never cached); a slow server with 8 concurrent readers (no block
fetched twice); bad URLs, 404, encoded bodies, non-HDF5 data. The cache
has unit tests for coalescing, splitting, LRU order, large reads,
failures and concurrent in-flight dedup.
ci-test.sh: clawhdf5-remote joins the no-C default-build check, and its
https feature is linted.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Version-2 B-tree (its writer's own node size and single-leaf layout),
Extensible Array and Fixed Array datasets written by FileBuilder, with
and without deflate, resized up and down along both dimensions and
written at random against a model; h5py, h5dump and h5rs check read the
result, and h5py resizes and rewrites every dataset afterwards.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
random_operations_match_a_model (edit_interop) now drives, on every
libver (earliest, v114, latest) and filter set (none, gzip + shuffle +
fletcher32, LZF):
- a dataset with one unlimited dimension and one with two (a version-2
B-tree chunk index under v114/latest), resized to random shapes that
shrink and grow any resizable dimension, with block and point writes;
- attributes on the first dataset under 20 names with values of random
types and sizes (scalars, int64 arrays, short strings, strings above
the heap's managed limit), so they move to dense storage on version-2
headers and are replaced by values of other sizes;
against a model where shrunk-away elements that come back read as the
fill value, compared with our reader and with h5py/numpy every 40 steps,
with h5dump and h5rs check; h5py then grows both datasets and adds an
attribute. The attribute check also compares h5py's attribute count with
libhdf5's object info. CLAWHDF5_EDIT_SEED reruns the workloads with other
random choices (seeds 1000-4000 pass).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
libhdf5 deletes a fractal heap's huge-object B-tree when the heap is
closed with no huge object left (H5HF__huge_term) and starts huge IDs
over. The editor left the empty tree, and a read-only libhdf5 then
failed to list the object's attributes: closing the heap tried to delete
the tree ("no write intent on file"), and h5dump failed the same way.
Replacing an object's last huge attribute (one above the heap's 4 KiB
managed limit) now deletes the tree (header and nodes freed), resets the
next huge ID and the wrapped flag, as libhdf5 does; a later huge
attribute creates a new tree. Bt2::delete frees a whole tree.
Found by the extended random-operation test; regression:
last_huge_attribute_replaced (fails before: h5dump, h5py listing), which
also compares the heap with libhdf5's after the same replacement.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
A FileEditor now keeps the space its edits free — a filtered chunk that
moved, chunks a shrink removed, B-tree nodes merged away, a heap's
replaced root indirect block or free-space section info, huge objects
replaced — and later edits allocate from it (best fit, lowest address
among equals, zeroed) before growing the file. An edit never reuses what
it frees itself: until it is committed the file still refers to that
space. Reused blocks are written in the commit's first phase with the
space past the old end of file (nothing on disk refers to them yet),
before any existing byte changes, so the crash-safety ordering holds.
Space still free when the editor is dropped is leaked, as libhdf5 leaks
it without a persistent free-space manager (files that have one, or use
paged aggregation, are still refused at open).
FileEditor::reusable_bytes reports what is left to reuse.
Tests: FreeList merging and best fit, the edit-local rule and the commit
split (image unit tests); a shrink followed by regrowth writing the same
data reuses every removed chunk and leaves the file size unchanged,
while one editor per edit grows the file, h5py/h5dump/h5rs check read
both and h5py continues (freed_space_is_reused_within_a_session).
measure_append_waste (edit_interop, ignored), same workload, one editor,
before -> after (bytes; libhdf5 in brackets), on tank 2026-09-26:
gzip chunks 1024, 1000 appends of 100: 307210 -> 306780 (306058);
gzip chunks 4096, 2000 appends of 10: 119684 -> 79829 (50292);
unfiltered unchanged (no space is freed).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
FileEditor::set_attr now handles every attribute storage libhdf5 uses
for version-2 object headers:
- objects that track (and index) attribute creation order: compact
attributes carry their creation index in the message header, the
Attribute Info message its maximum;
- the move to dense storage when an object reaches its compact limit (or
an attribute is too large for a header message), as H5O__attr_create
does it: a new fractal heap, name index (v2 B-tree type 8) and, when
creation order is indexed, creation-order index (type 9); the compact
attributes moved over in header message order, their messages freed;
- objects already in dense storage (h5py- or clawhdf5-written): new
attributes inserted (H5A__dense_insert), an attribute replaced by one of
the same encoded size rewritten in its heap object (H5A__dense_write),
otherwise removed from both indexes and the heap and inserted anew.
edit/fheap.rs follows H5HF: managed objects go to the best-fitting free
section of the heap's free-space manager (FSHD/FSSE, kept as libhdf5
keeps it — sorted sections, counts, section info reallocated when its
size changes, the manager deleted when empty); otherwise to a new direct
block: the root direct block of an empty heap, else the block at the
allocation iterator in the root indirect block (created from the root
direct block, doubled as needed), with libhdf5's managed-space, allocated
space, free space and iterator bookkeeping. Objects above the managed
limit are huge objects in their own space, indexed by the huge-object
B-tree (type 1). Removed objects return their space merged with adjacent
free space. Refused before anything is written: heaps with I/O filters,
child indirect blocks, an object larger than the next heap block
(libhdf5 skips blocks and records them as free space), free sections
other than those inside direct blocks, removing a direct block's last
object (libhdf5 frees the block), directly addressed huge objects.
Attributes are encoded as libhdf5 does when h5py opens a file r+ (low
bound "earliest"): message version 1 (3 for non-ASCII names), simple
dataspaces with their maximum dimensions. Header chunks are now visited
in libhdf5's order (FIFO), which is also the order attributes move to
dense storage in.
Tests (edit_coverage_interop): 40 attributes on each of a plain group, a
group tracking and indexing creation order, and a dataset (earliest,
v110, latest), some above the 4 KiB managed limit, then same-size
rewrites: the heap statistics, free-space sections and both index
B-trees node for node equal libhdf5's doing the same through h5py; then
replacements of other sizes, h5py adds/deletes/rewrites; h5py, h5dump,
h5rs check and our reader agree throughout, h5py's attribute count
included. clawhdf5-written dense storage (tracked and untracked) is
extended the same way; refusals leave the file byte for byte as it was.
edit_interop's attribute test now expects dense storage and tracked
creation order to work.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
libhdf5 follows any number of continuation chunks, and a header that is
full gains one per message added (each new chunk holding the next
continuation message), so a version-1 header with a few dozen attributes
added one at a time is a chain dozens of chunks long. The reader recursed
once per chunk and refused a chain deeper than 32 (NestingDepthExceeded):
h5py read such files, we did not. Version-2 headers stopped at 256
continuation chunks.
Version-1 chunks are now followed with an explicit stack (the same
depth-first message order as before), version-2 ones as before; both
refuse a chunk address seen twice (a cycle, what the limits guarded
against) and more than 65 536 chunks.
Regression: long_v1_continuation_chains_are_read (a 200-chunk chain),
v1_continuation_cycles_are_refused; the dense-attribute interop test's
'earliest' case produces such a chain.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
CHANGELOG (Unreleased): File::open_storage, raw data and v2 B-trees over
Storage, the tests and their corpus results (2026-09-26, tank; conformance
600 of 697, results.json identical to 8f59b2e). known-issues: what
open_storage does not do yet (no remote backend or block cache, read_at
counts of a one-pass read, v1 group lookups, whole-file VDS sources,
zero-copy methods, SWMR growth, hash-order error choice on damaged chunked
datasets). Design: M2 status and the choices made.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
File::open_storage(Arc<dyn Storage + Send + Sync>) opens a file served by
any backend: groups, datasets, attributes, read_*, selections, VL data
and virtual datasets (external sources through the new
File::set_vds_resolver) all read through Storage::read_at/read_ranges.
The file's view (user block skipped, bounded by the recorded end of
file) is itself a Storage; File::open and File::from_bytes keep their
mmap and in-memory paths, now as that view's as_contiguous() fast path.
A storage-backed file's metadata cache image is laid over each read it
covers (new CacheImage::entries), as libhdf5 loads it.
The typed readers keep their fast paths over any storage: a contiguous
dataset is read in one piece and converted (read_f64 and friends), and a
contiguous native selection reads only its runs
(data_read::read_selection_native_in). Zero-copy methods answer
ContiguousStorageRequired when the bytes are not in memory, and
File::as_bytes panics there (File::contiguous_bytes is the fallible
form).
Test: tests/storage_equivalence.rs reads every fixture, and with
CLAWHDF5_STORAGE_CORPUS every conformance-corpus file, through File::open
and through open_storage over a read_at-only CountingStorage — tree,
attributes, and every dataset's values several ways — and requires
identical transcripts; it prints the read_at calls and bytes a one-pass
read costs per file.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
FileEditor can now:
- add, move and resize chunks of datasets with two or more unlimited
dimensions (version-2 B-tree chunk index, record types 10/11). The new
edit/btree2.rs follows libhdf5's H5B2 code: H5B2_update (modify, or
insert into a leaf with room, or fall back to H5B2__insert), the
preemptive split/redistribute loop with its two retries, split1,
split_root (depth growth, node geometry per depth), redistribute2/3,
cumulative record counts and the pointer widths H5B2__hdr_init derives,
a checksum per node. Removal (H5B2_remove: merge2/3, redistribution,
root collapse, the internal-record swap with a leaf) is there too. A
dataset without an index yet gets one from the layout message's node
size and split/merge percentages.
- shrink a chunked dataset along any dimension (resize to a smaller
shape), as H5D__set_extent / H5D__chunk_prune_by_extent do: the same
chunks visited in the same order; chunks wholly outside the new extent
are removed from the index (version-1 B-tree: H5B_remove with its
sibling key and link fix-ups and the empty-root case; version-2
B-tree; Fixed/Extensible Array elements reset to the fill element;
an implicit index keeps its chunks, as libhdf5 does) and their space
noted as free; the part of each partial edge chunk outside the extent
is overwritten with the fill value, so elements that come back after
a later growth read as fill.
- under early allocation, allocate and fill the chunks a growth brings in
(H5D__chunk_allocate), which an implicit index needs: libhdf5 refills
them, and they may hold the data of chunks pruned earlier.
Tests (crates/clawhdf5-tools/tests/edit_coverage_interop.rs): growth in
both dimensions of v110/latest files, unfiltered and deflated, gives
node-for-node the version-2 B-tree libhdf5 builds (h5py with its chunk
cache off, so chunks enter the index in the editor's order), through a
depth increase; random chunk order; 60 random shrink/grow/write steps on
Extensible Array, version-2 B-tree, Fixed Array, 1-D and implicit
datasets (earliest/v110/latest, with and without gzip+shuffle) give the
values h5py gets doing the same and the same index shape (version-1 and
version-2 B-tree node shapes, Extensible Array statistics); h5py r+
continues on every result; h5dump and h5rs check accept them.
edit_interop's version-2 B-tree case now appends instead of expecting a
refusal; shrinking is no longer an error in edit_tests.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
libhdf5 allocates an implicit index's chunks for the whole maximum
extent and places chunk `scaled` at its row-major position in the
maximum chunk grid (H5D__none_idx_get_addr, max_down_chunks). The reader
used the current grid, so a dataset below its maximum shape with more
chunk columns at its maximum read other chunks' values from the second
chunk row on (h5py early allocation, fixed maxshape).
generate_implicit_chunks_in_grid takes the maximum dimensions;
generate_implicit_chunks keeps its signature (grid = current extent).
Regression: implicit_chunks_use_the_maximum_grid here, and
implicit_index_below_its_maximum_reads_like_libhdf5 (h5py file) with the
editor tests.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Every raw-data path has a generic *_in core, with the &[u8] functions as
thin wrappers: data_read (read_raw_data*, read_raw_data_selection,
read_chunked_native), chunked_read (the v1 B-tree chunk index, list_chunks,
the full, cached, sweep and indexed reads), parallel_read, partial_read,
fill_value (read_full_with_fill, apply_to_unallocated_chunks; and
dataset_fill_value_from_storage is now generic), vds (the virtual file
through Storage, external sources still through the resolver),
vl_data (VlResolver<'a, S = [u8]>, read_vl_strings_in, read_vl_bytes_in),
AttributeMessage::read_vl_strings_in and provenance::verify_dataset_in.
With the whole file in memory nothing changes: chunks and contiguous data
are sliced from it as before. Otherwise a chunked read lists its chunks,
fetches their stored bytes with one Storage::read_ranges call per 64 MiB
batch (chunks the cache already holds are not fetched), then decodes as
today; a selection fetches only the chunks it overlaps, and a contiguous
selection only its runs. Each extent's bounds error is the one the slice
code gave, reported when that extent is reached, so errors keep their
order.
Tests: the equivalence harness now reads every dataset's values (whole,
fill-aware, cached, indexed, three selections, VDS, VL strings and
sequences) through the read_at-only storage and requires the slice
results (all 653 corpus files agree); a misbehaving storage (a failing
Nth read, short reads) only ever yields errors or the right values; and
chunked reads are checked to use one read_ranges call.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
BTreeV2Header::parse_in, collect_btree_v2_records_in and
find_btree_v2_records_in read one bounded window per node (its size is
known from the parent before the node is read; a count stretched past
node_size is checked against the end of the file first), with the
whole-file bounds errors unchanged. With them, dense attributes, a SOHM
B-tree index and huge fractal-heap objects no longer answer
ContiguousStorageRequired, and group_v1/group_v2 listings, lookups and
path resolution get *_in cores (resolve_group_children_in,
resolve_child_in, resolve_path_any_in, ...). The &[u8] functions are
thin wrappers, as in M1.
The equivalence harness now fails on any ContiguousStorageRequired and
compares v2 B-tree headers, records and descents, group listings, child
lookups and paths; a unit test compares a two-level tree through a
read_at-only storage truncated at every length and with every node byte
flipped.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
+6-7% (about 4 ns per header) is real; symbol-table nodes -17%, group
B-tree walk -16%, facade listing -2.4%: local metadata reads are net
slightly faster than main.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Regenerated on tank: ok 599 -> 600 (h5ex_d_zfp.h5), our-error 4 -> 3,
mismatch 2, no panics, hangs, crashes or OOM. Baseline raised.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
FileBuilder stored every chunk of an LZF or Blosc dataset through the
filter with filter mask 0. libhdf5 counts LZF and Blosc output no smaller
than the chunk as a failure of the optional filter and stores the chunk raw
with the filter's mask bit set. For an LZF chunk whose stream was exactly
the chunk's size, the first libhdf5 rewrite stored raw data at the same
size and kept our stale mask 0 in the index, and h5py could no longer read
the dataset.
precompress_chunks now runs chunks through compress_chunk_masked (as
FileEditor does since f7e2ab1), sequentially and on the parallel path, and
build_chunked_data_from_precompressed records each chunk's real mask in
every index the writer builds: single chunk (layout field), Fixed Array and
Extensible Array filtered elements, and version-2 B-tree type 11 records
(create_datasets_parallel goes through the same path). The writer builds
no version-1 B-tree or implicit index. PrecompressedChunks::chunks gains
the mask. Files whose chunks all compress are byte-identical.
Latent only in the unreleased LZF/Blosc writer (added 2026-09-26); no
tagged release writes either filter.
Tests:
- plugin_filters_interop skipped_optional_filters_are_masked_as_libhdf5_masks_them:
LZF, shuffle+LZF+fletcher32 and Blosc over random, compressible and
alternating chunks in every index; masks equal an h5py-written twin's;
h5py r+ rewrites and extends them; h5py, h5dump and our reader read
every value. Before: 20 of 24 datasets had masks other than h5py's, and
with that check disabled h5py failed to read the rewritten datasets
("filter returned failure during read").
- plugin_filters_interop files_whose_chunks_all_compress_are_unchanged:
pins the pre-fix bytes of five all-compressing files.
- chunked_write skipped_lzf_chunks_are_masked_in_every_index (fails before:
mask 0, want 2).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
- Per-file probe output is identical for 696 of 697 files, not all:
cve-2025-2310.h5's error string depends on which parallel chunk decode
fails first, at f2ff2c4 as on this branch.
- The parser cores are generic (S: Storage + ?Sized); provisional A/B
numbers against f2ff2c4, including the one bench that still shows
ObjectHeader::parse slower when old and new are separate binaries.
- Reads sized by untrusted fields are bounded; the harness only accepts
the known whole-file fallbacks.
- range-reads.md records why M1 went generic rather than &dyn.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
On a backend without the file in memory, a structure read whose length
comes from untrusted header fields was clamped only by the end of the
file, so a crafted size made one read (and copy) of up to the rest of
the file. Each such read now covers what the parser actually uses:
- local heap names: read in growing pieces (64 bytes first, then 4x)
up to the end of the data segment, instead of the rest of the segment
per name (quadratic for a big symbol-table group);
- fractal heap indirect blocks: the doubling-table geometry locates the
entry covering the object, and the first read ends at that entry; only
if it is unallocated does the walk read the rest of the block (it
visits every entry then). One walk implementation serves both;
- paged fixed/extensible array data blocks over 1 MiB: the prefix and
page bitmap, then each page in use on its own (smaller blocks are
still one read);
- blocks under one checksum (non-paged array data blocks, extensible
array index and super blocks): the bounds check that comes first (the
checksum's; the page bitmap's for a super block) is made against the
file length before reading (Window::check_extent), so a block claimed
past the end of the file costs no read. With the checksum feature off
the parser has no such first check and the old read stands.
Other windows were already bounded (the superblock and object header
prefixes, the fractal heap header by a u16, SOHM tables by u8/u16
counts) or are exact reads checked against the file length first.
In memory nothing changes: the pieces are borrowed slices.
Tests: CountingStorage over a crafted heap (16 MiB file, width and rows
0xFFFF: under 1 KiB read, 16.7 MB before), a heap segment claiming 64 MiB
(one 64-byte read per short name), long names at every piece boundary,
a fixed array block claimed past the end of a 16 MiB file (under 64
bytes read), and in the equivalence harness an h5py file with a 2.4 MB
fixed array block and a >1 MiB extensible array block, whole and cut at
97 points: every chunk index agrees with the slice read and the largest
takes 205 KB (2.4 MB and 1.2 MB when read whole).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The harness counted ContiguousStorageRequired from any parser as an
allowed difference, so a converted module that wrongly fell back to the
whole file would still pass. It now accepts the error only from the
three sites that are not converted yet (dense attribute storage, a SOHM
B-tree index, huge fractal-heap objects, all found through a v2 B-tree)
and only in the checks that can reach them; anything else fails with
the check and the site named.
Checked by making LocalHeap::parse_in return the error first: the
fixture and h5py runs fail ("local heap fell back to the whole file").
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Every `*_in` core and the read helpers take `file: &S` with
`S: Storage + ?Sized` instead of `&dyn Storage`, and the `&[u8]`
wrappers pass the slice itself, so they compile to a `[u8]` instance:
`as_contiguous()` inlines to `Some(self)` and each structure read is the
slice code's bounds check again, with no indirect call. `&dyn Storage`
still works (`S = dyn Storage`); there is one parser implementation.
Also, so the structure reads cost no more than the slice checks did:
- ObjectHeader::parse_in reads the prefix once (signature included)
instead of the signature and then the prefix: two reads for a
one-chunk header instead of three on a range backend;
- the symbol-table node and group B-tree (v1) loops walk their entries
with chunks_exact over the bytes read, and the node's redundant second
bounds check is gone (the entries' read is the check, same error);
- a version-1 header's message list is sized from its (capped) count.
Same results and errors; the unit and equivalence tests are unchanged.
New Criterion bench `clawhdf5/benches/local_metadata_bench.rs` over a
400-group version-1 file written by h5py (new fixture
`v1_groups_400.h5`): ObjectHeader::parse, symbol-table nodes, the group
B-tree walk and a facade listing, using only APIs that exist at f2ff2c4
so it builds there for an A/B.
Provisional A/B against f2ff2c4 (busy machine, not for docs): both
builds linked into one binary and timed in alternation, 200 rounds;
median ratio new/old: facade listing -0.5% to -3.5% (was +14%),
ObjectHeader::parse +1% to +2% (was +25%), symbol-table nodes -18%,
group B-tree walk -18%, local-heap names and resolve_group_children
within +-1.5%. An old-vs-old-copy run shows +-2% from code layout alone.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Each edit planned over the reader's memory map of the file and committed
while that File, and the Image's &[u8] over the mapping, were still alive,
writing the same file through the editor's descriptor. Nothing read the
mapping during the writes, but a shared slice whose memory changes
underneath it is undefined behaviour under Rust's aliasing rules.
Image::into_plan now detaches the edit's writes (patches, end of
allocation) into a Plan that owns all of its bytes and borrows nothing;
edit() takes the user-block size, drops the File — unmapping the file —
and only then commits the Plan. The invariant is documented in the image
module and the editor's module docs.
Test: edit::tests::file_is_not_mapped_while_an_edit_writes_it checks
/proc/self/maps at the moment each commit starts (write, resize, set_attr):
never mapped. With the commit moved back before the reader is dropped (the
previous order) it reports all three commits with the file mapped.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
A dataset whose filter this build cannot encode (scale-offset, N-Bit, SZIP;
a plugin filter the build lacks) failed with Error::Format("unsupported
filter: 6"), although the editor documents every refused edit as
Error::Unsupported, and the Python bindings raised ValueError rather than
NotImplementedError. Every edit now maps FormatError::UnsupportedFilter to
Error::Unsupported; the file is left untouched as before.
Test: edit_interop unencodable_filters_are_unsupported — h5py scale-offset
datasets (integer with chunks, integer never written, float D-scale):
Error::Unsupported naming the filter, and the file byte for byte unchanged.
Fails on the previous editor (Format(UnsupportedFilter(6))).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
libhdf5 counts a version-2 object header's attributes through its Attribute
Info message (0x15) and reports none when the header has none. set_attr gave
v110/latest groups, the root group and datasets without attributes an
attribute message only, so h5py listed the attribute but len(obj.attrs) and
H5Oget_info's num_attrs said 0, and stayed wrong after h5py r+ added more.
Like H5O__attr_create, the edit now adds the message when a version-2 header
lacks it, in the same planned edit: version 0, the header's creation-order
track/index flags, maximum creation index 0, undefined fractal heap and
B-tree addresses, message flag DONTSHARE — byte for byte what libhdf5
writes. It goes before the attribute (libhdf5's order) when free space
holds both, else after it, so a continuation chunk made for the attribute
also takes it.
Test: edit_interop attribute_count_in_version_2_headers — v110 and latest
files, attributes set on the root group, groups and datasets with and
without existing attributes: h5py's len/num_attrs/list/values, h5dump -A
and our reader agree, also after h5py r+ adds attributes up to and past the
compact limit. Fails on the previous editor (h5py len 0).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
A valid group has one link per name, but a damaged or hand-made one can
have two. resolve_child followed the first soft link of the name, the
listing skipped a dangling one and listed the name via a later link, and
path resolution followed the last symbolic link: three answers. All now
take the first link of the name (header message order in a compact group,
name index order in a dense one) and ignore the rest, even if the first
dangles. That is libhdf5's rule for compact groups (H5G__compact_lookup
stops at the first Link message); h5py opens nothing for a dangling first
link although a later one resolves. For a dense group libhdf5
binary-searches the index and may land on another of several exact
duplicates; documented on first_link_named. find_symbolic_link's v2 branch
was dead (only v1 groups reach it) and is now v1-only.
Test: an h5py compact group with soft links dup_A (dangling, or to /d) and
dup_B (the other), dup_B renamed to dup_A in the header and re-checksummed.
Lookup, path and listing through all three readers match h5py for both
orders. With the old group_v2.rs the path lookup returned 42 where h5py
opens nothing.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
addr::to_usize's error branch only ran where usize is narrower than u64,
and no such target runs tests in CI, so on x86_64 the test checked only
that every u64 fits. to_usize and saturating_usize are now the usize
instances of generic to_index/saturating_index; the test runs the same
code with u32 standing in for a 32-bit usize: values past u32::MAX
(including one an `as` cast would wrap to 0x1234) are Overflow, and the
saturating form clamps. A mutant that truncates instead fails the test;
the old addr.rs does not provide the helper the test needs.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The editor stored every chunk through the whole pipeline with filter mask 0.
For LZF that did not shrink a chunk, h5py instead stores it raw with the
filter's mask bit set. A chunk the editor stored LZF-encoded at exactly the
raw size was then rewritten raw by libhdf5 at the same size; libhdf5 does not
touch the index entry when the size is unchanged, so the stale mask 0 stayed
and h5py (and h5dump) could no longer read the dataset.
clawhdf5_format::filters::compress_chunk_masked runs the pipeline as
H5Z_pipeline does: an optional filter (H5Z_FLAG_OPTIONAL) that fails is
skipped and its bit set, a mandatory one fails the write, and LZF/Blosc
output no smaller than the input counts as failure, as in the reference
filters (their output buffer is the input's size). Deflate, LZ4, Zstd,
bitshuffle and bzip2 never fail on size in libhdf5 and are kept as before.
Test: edit_interop optional_filters_that_fail_are_skipped — the reviewer's
repro at every libver: the editor stores the chunk exactly as h5py does
(mask 1, size 5; shuffle+LZF+fletcher32 mask 2), h5py r+ rewrites and
extends the datasets, and h5py, h5dump and our reader read every value.
Fails on the previous editor (mask 0; h5dump cannot read /u8).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Three `chunk_info.address as usize` casts behind the `parallel` feature
survived the conversion, because check-32bit-casts.sh linted only default
features plus plugin-filters. On a 32-bit target with rayon a chunk address
past 4 GiB still wrapped onto another part of the file. They go through
addr::to_usize now, and the lane index (h % n, always < n) through
saturating_usize.
The script now lints no default features, default features, and every
optional feature but szip (wasm32; the set with zstd, which does not build
for wasm32, on the host, where the lint reports the same casts). With the
old parallel_read.rs/lane_partition.rs it fails listing the four casts; the
old script passed them. CHANGELOG and the design note give the exact count
(119) and what is not covered.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Only leaves and the header were checked. Harmless while every lookup read
the whole tree, but the indexed lookup prunes children by the keys stored
in internal nodes, so one corrupted byte there could route a name to the
wrong child and report it missing with no error. A BTIN whose lookup3
checksum does not match is now ChecksumMismatch on every read (lookups and
full traversals), as in libhdf5.
Test: one byte of the root BTIN of the 35 001-link h5py group's name index
changed -> lookups, paths and listings through File, MmapFile and LazyFile
all fail with ChecksumMismatch, and h5py refuses both. Before, lookups
returned Ok.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
read_exact_at and read_upto (and so Window::read) ask as_contiguous()
first and slice the file directly when the backend holds it in memory:
one dynamic call per structure read instead of two or three (len,
read_at, then len again for errors). Same results and errors.
Provisional (busy machine, not for docs): a listing that walks 400
symbol-table groups through the facade went from about 18% to about 14%
slower than before the Storage conversion; the extra cost is a few tens
of nanoseconds per structure read, which the facade's per-lookup
re-listing (range-reads.md M0) multiplies.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
known-issues records what the editor refuses, that freed space is never
reused (append-workload file sizes measured 2026-09-26 on tank with the
ignored measure_append_waste test; sizes are deterministic), and that there
is no journal.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
New clawhdf5::FileEditor opens an HDF5 file (h5py-written at any libver,
HDF5 2.0 format included, or clawhdf5-written) under an exclusive flock
and changes only what an edit touches:
- write_selection/write_all/write_values: compact, contiguous (also
late-allocated) and chunked datasets, any selection. Chunks are decoded,
updated and re-encoded; a filtered chunk that no longer fits moves to the
end of the file unless it is the file's last structure, which grows in
place. New chunks go into v1 B-tree, Extensible Array (paged data blocks
included), Fixed Array and single-chunk indexes, created on first use.
- resize: grow chunked datasets up to maxshape.
- set_attr: add/replace compact attributes, in a NIL slot or a new
continuation chunk.
Each edit is planned in an in-memory image and refused whole
(Error::Unsupported) when any part is unsupported (v2 B-tree / implicit
new chunks, shrinking, vlen/reference data, dense or order-tracked
attributes, cache images, paged/persistent free space). Commit writes and
syncs new space before patching existing bytes. Layout v5 (HDF5 2.0)
array indexes use 8-byte filtered chunk sizes, as libhdf5 does.
Error gains Unsupported/InvalidArgument/Locked and is #[non_exhaustive];
the Python bindings map them. build_attr_message is public.
Tests (h5py, h5dump, h5rs check --data after every round; h5py r+
afterwards): appends crossing EA super/data blocks and B-tree splits, the
same B-tree node counts and EA statistics as libhdf5 for the same writes
(in order, reversed and shuffled; paged blocks), every layout and chunk
index overwritten under random selections, attributes to continuation
chunks, random operations against a model, refused edits leave the file
byte-identical, locking.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Every `u64 as usize` cast in clawhdf5-format (115 on wasm32) now goes
through addr::to_usize for values read from the file — addresses, lengths,
counts, dimensions: FormatError::Overflow where the value does not fit
instead of wrapping onto another part of the file on a 32-bit target — or
addr::saturating_usize for counts bounded by something in memory (codec
progress counters, writer sizes), which fail a bounds check or allocation
rather than wrap. A chunk whose offset does not fit lies outside the
dataset and is skipped; partial reads treat such an offset as out of the
buffers. On 64-bit targets nothing changes.
scripts/check-32bit-casts.sh (run by ci-test.sh) lints the wasm32 build
with clippy's cast_possible_truncation and fails on any u64 -> usize
finding; before this commit it listed 115.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Finding one link or attribute by name read every entry: Group::dataset and
Group::group (File, MmapFile, LazyFile) listed the whole group per call, and
path resolution scanned each group's links. Opening every child of a
35 001-link group by name decoded ~1.2e9 links.
Now a dense group's v2 B-tree name index (type 5, lookup3 hash of the
name) is descended to the records with the name's hash
(btree_v2::find_btree_v2_records reads only the nodes whose key interval
overlaps), and only those links are read and compared; all hash-equal
records are compared, so libhdf5's tie order does not matter. Dense
attributes the same through their type 8 index
(attribute::find_attribute_in_file, facade attr(name)); huge heap objects
through their ID-ordered index. group_v2::resolve_child returns what the
listing has under a name (soft links followed, dangling/external ones not
found). Group::entries and File::group_at hand out a listing's addresses.
The lookup-stats feature counts heap objects read. Tests: one lookup in
an h5py-written 35 001-link group with colliding hashes reads at most two
links (before: 35 001, failing), attribute lookups likewise (before: 3 000,
failing), every child opens through all three readers and matches h5py,
every link kind resolves as h5py resolves it in dense and compact groups,
300 huge attributes are found, and a range search matches a full scan at
every tree depth.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Lists the parsers now reading through Storage, what still needs the
whole file (v2 B-tree-indexed structures: a clean error; raw data: M2),
the equivalence harness, and the evidence that nothing changed: existing
tests, a byte-identical conformance results.json and per-file probe
output against f2ff2c4, and identical slice-API transcripts over 748
files between the two builds.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
With Storage in scope, `data.len()` on a `&&[u8]` resolves to
Storage::len (already a u64), so `as u64` was a no-op cast; name the
slice method explicitly.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Making dataset_fill_value_in generic over Storage broke callers that pass
an array (`include_bytes!`): a generic parameter does not unsize-coerce
`&[u8; N]`. It takes &[u8] again, as before this branch, and wraps the
new dataset_fill_value_from_storage(&dyn Storage, ..).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
tests/storage_equivalence.rs walks real files and runs every metadata
parser converted to Storage twice per object — over the file as a slice
and over a read_at-only CountingStorage — and requires identical results,
values and errors alike; the only allowed difference is the clean
ContiguousStorageRequired from the structures still indexed by a v2
B-tree (dense attributes, a SOHM B-tree index), which is counted. It
covers superblock and extension, cache image, SOHM table/list/B-tree,
object headers, attributes, fill values, shared messages, symbol-table
groups (local heap, B-tree, nodes, names), fractal heaps and their
objects, VDS mappings, and fixed/extensible array chunk indexes.
Inputs: every fixture; files h5py writes for what the fixtures lack
(extensible arrays with super blocks and paged data blocks, paged fixed
arrays, a 400-group v1 file with a user block, a 300-link dense group,
dense, shared and committed-type attributes, SOHM list and B-tree
indexes; honours CLAWHDF5_PYTHON / CLAWHDF5_REQUIRE_INTEROP); and, with
CLAWHDF5_STORAGE_CORPUS set, a corpus such as conformance/.cache/corpus.
Milestones M2/M3 add their parsers to check_object.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
ZFP (32013) was the one plugin filter still listed as UnsupportedFilter.
Conformance on tank, `conformance/run.sh --no-fetch` (2026-09-26): 600 of
697 files ok (baseline 599); h5ex_d_zfp.h5 is newly ok.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
DataLayout::resolve_vds_mappings_in reads the global heap collection
holding a virtual dataset's mappings through GlobalHeapCollection::
parse_in; resolve_vds_mappings is a wrapper. The rest of the data layout
module parses message payloads and does not read the file.
New test: the virtual dataset of vds_same_file.h5 resolves to the same
mappings through a read_at-only CountingStorage, in two reads.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
dataset_fill_value_in now takes the file as any Storage (generic, so
every &[u8] caller compiles unchanged) and follows a shared fill value
message through message_data_with_sohm_in. The raw-data helpers in the
module (read_full_with_fill, apply_to_unallocated_chunks) walk chunk
indexes and stay on &[u8] until milestone M2.
New test: the four datasets of shared_fill_value.h5, two with their
fill value in the SOHM heap, resolve identically through a read_at-only
CountingStorage.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
AttributeMessage::parse_in_storage, extract_attributes_full_in and
extract_attributes_tolerant_in take the file as &dyn Storage: shared
datatypes, dataspaces and attributes are resolved through the Storage
shared-message path, and dense attributes' fractal heap through
FractalHeapHeader::parse_in / read_managed_object_in. The dense-storage
name index is a v2 B-tree, which is not read over Storage yet: over a
backend without the whole file in memory it is a clean
ContiguousStorageRequired error, never a partial list. The &[u8]
functions are wrappers.
New test: every object in five h5py-written fixtures (compact, shared
and dense attributes) reads identically through a slice as Storage, and
through a read_at-only CountingStorage except the dense ones, which give
the clean error.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Every shared-message entry point has an *_in(&dyn Storage, ..) core:
message_data(_with_sohm), resolve_shared_message(_with_sohm),
resolve_sohm_message, load_sohm_table, and the SMTB/SMLI parsers, which
read the signature and then all entries in one bounded read (the list's
open-ended &file_data[pos..] slice is gone). Object headers and the SOHM
fractal heap are read through their Storage parsers; a SOHM B-tree index
still needs the v2 B-tree over a slice, so over a backend without one it
is a clean ContiguousStorageRequired error. New signature::find_signature_in
probes the candidate offsets with 8-byte reads. The &[u8] functions are
wrappers.
New tests: SOHM tables and lists with 4- and 8-byte offsets, at two
offsets, cut at every length and with a bad signature, parse identically
through a read_at-only CountingStorage in at most two reads; the
signature search matches the slice search.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
ExtensibleArrayHeader::parse_in reads the header in one read, and
read_extensible_array_chunks_in reads each index block, super block and
data block as a prefix read and then one window of the whole structure
(paged data blocks included); checksums and elements are checked in the
window with bounds errors reported as the whole-file checks did. The
&[u8] functions are wrappers. New test: an array with inline elements
and a data block, cut at every length and damaged in each structure,
reads identically through a read_at-only CountingStorage.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
FixedArrayHeader::parse_in reads the header in one read, and
read_fixed_array_chunks_in reads the data block's prefix and then the
whole block, paged or not, as one window; checksums and elements are
checked in it, with bounds errors reported as the whole-file checks did
(also in builds without the checksum feature, where the per-element
checks are the only ones). The open-ended &file_data[offset..] slices are
gone. The &[u8] functions are wrappers. New test: non-paged and paged,
filtered and unfiltered arrays, cut through the data block and with
damaged bytes, read identically through a read_at-only CountingStorage.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The facade forwards `zfp` and adds it to `plugin-filters`.
tests/zfp_interop.rs: h5py + hdf5plugin (H5Z-ZFP 1.1.1, zfp 1.0.1)
write 2205 datasets over 16 modes (rate, precision, accuracy,
reversible, expert settings at their edges) x int32/int64/float/double
x 1-4-D shapes with partial edge chunks, partial blocks and unit chunk
dimensions x smooth/noisy/wide-range/zero/inf-NaN data; clawhdf5 must
read each byte for byte as h5py does (read back after closing the file:
h5py returns a chunk still in libhdf5's cache without decoding it). A
second test swaps a file's header words to what a big-endian writer
stores and checks the byte-swapped values match h5py's.
The left-out-filter test now expects ZFP only in builds without it; CI
lints `zfp` alone and runs the new test with the plugin-filter step.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
A port of the zfp 1.0.1 decoder and of H5Z-ZFP 1.1.1's decompression
path behind a new `zfp` feature (in `plugin-filters`): every mode
(fixed rate, precision, accuracy, reversible, expert), int32, int64,
float and double, 1-4 dimensional fields with partial blocks, and
headers written big-endian (values byte-swapped as H5Z-ZFP does).
H5Z-ZFP keeps the zfp header in cd_values (version word, then the
magic/metadata/mode bit stream); each chunk is the bare stream. The
decoder reproduces libzfp bit for bit: integer arithmetic wraps as
libzfp's, block exponents scale by exact powers of two, and the mode
goes through zfp_stream_mode as H5Z-ZFP hands it to zfp. A stream that
ends early is an error (libzfp reads past its buffer), as is a field
whose size is not the chunk's; the output is allocated only once the
stream holds a bit per block.
Tests: header/mode unit tests, and a counting-allocator fuzz of random
headers (expert parameters at their edges) and streams.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
FractalHeapHeader::parse_in reads the header as one window (a second,
longer one when it holds an I/O filter pipeline); read_managed_object_in
reads direct blocks, indirect blocks (one window up to the last child
entry) and huge objects with bounded reads. The &[u8] methods are
wrappers. A huge object indexed by the huge-object v2 B-tree, which is
not converted yet, is a clean ContiguousStorageRequired error on a
backend without the whole file in memory (after the "no index" check,
so the error order is unchanged).
storage::Window (crate-internal) reads a window of a structure and
reports bounds failures exactly as the whole-file ensure_len did, and a
short read inside the file is now a Storage error rather than an EOF.
New tests: headers (with and without a filter pipeline) cut at every
length, and managed objects in a direct root and through an indirect
root, huge objects with direct IDs and tiny objects, give identical
results through a read_at-only CountingStorage.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
BTreeV1Node::parse_in reads a node's header and then its keys and
children, two bounded reads; collect_symbol_table_nodes_in walks the tree
over any Storage. The &[u8] functions are wrappers. New test: nodes with
siblings and 4- and 8-byte offsets cut at every length, and a two-level
tree with truncated leaves, give identical results through a read_at-only
CountingStorage (six reads for the three nodes).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
SymbolTableNode::parse_in reads the node's 8-byte header, checks the
entries against the end of the file as before, and reads them in one
bounded read. parse is a wrapper. New test: nodes with 4- and 8-byte
offsets, valid and with a bad version, at two offsets and cut at every
length, parse identically through a read_at-only CountingStorage.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
GlobalHeapCollection::parse_in / parse_index_in read the header, check
the collection against the end of the file, and read the collection in
one bounded read; objects are indexed in it with file offsets, as before.
The &[u8] functions are wrappers. New test: collections with 4- and
8-byte lengths, one whose size runs past the file and one whose object
overruns it, at two offsets and cut at every length, give identical
results through a read_at-only CountingStorage.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
LocalHeap::parse_in reads the header in one bounded read,
validate_free_list_in reads each free block's two lengths, and
read_string_in reads from the string to the end of the data segment
once and looks for the terminator there. The &[u8] methods are wrappers.
New test: a heap without free space, with a valid free block and with a
free block overrunning the segment, cut at every length, parse, validate
and read strings identically through a read_at-only CountingStorage.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
read_superblock_extension_in, cache_image_state_in, CacheImage::decode_in
and CacheImage::block_in take &dyn Storage (whose length is the end of
file); the image block is one bounded read. The &[u8] functions are
wrappers; applying an image in place still needs the bytes in memory.
New test: extension messages, a cache image and a corrupt one decode to
the same results through a read_at-only CountingStorage.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
ObjectHeader::parse_in(&dyn Storage, u64, ..) reads the signature, the
prefix (a window of at most 34 bytes for version 2) and then each chunk,
continuation chunks included, as one bounded read; the message loops run
unchanged on the chunk with chunk-relative positions. parse keeps its
&[u8] signature as a wrapper. Bounds errors are reported as before, with
absolute positions and the file's length. New test: headers of both
versions, with times, phase-change values, creation order and a
continuation chunk, and every truncation of each, parse identically
through a read_at-only CountingStorage.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Superblock::parse_in(&dyn Storage, offset) reads one bounded window of
128 bytes (the largest superblock is 100) and runs the existing version
parsers on it; parse and refresh_eof keep their &[u8] signatures as
wrappers. No behaviour change: on a file longer than the window no bounds
check can fail, and on a shorter one the window is the whole file. New
test: every version and truncation parses to the same result through a
read_at-only CountingStorage, in one read.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Range-read milestone M1, first step (docs/design/range-reads.md §3(a)):
a synchronous, no_std read interface with u64 offsets, read_at returning
Cow<[u8]>, read_ranges, len and an as_contiguous fast path. Implemented
for [u8], Vec<u8>, &T, Box<T> and Arc<T>; slices serve borrowed bytes.
read_exact_at reproduces the parsers' UnexpectedEof bounds error exactly,
so converted modules keep their error values.
FormatError gains Storage(String) and ContiguousStorageRequired; it and
the facade Error are now #[non_exhaustive] (breaking for exhaustive
matches, noted in the changelog; the Python bindings' match gets a
wildcard arm).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
addr::to_usize turns a 64-bit file address or length into a slice index,
failing with FormatError::Overflow where it does not fit usize (32-bit
targets such as wasm32) instead of truncating like an `as usize` cast.
Callers are converted in the following commits.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Idle-start run on tank at c5334b1 (noisier than the last: compare ratios
within the run). Full reads of deflate data at 16 threads: 4944 MB/s vs
3135 for 16 h5py processes (1.58x; 0.69x-0.76x before). One thread with
the default pool: 6143 MB/s, 15x one h5py call. The concurrent-read
known issue is closed.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Regenerated on tank: ok 575 -> 599, our-error 10 -> 4, mismatch 20 -> 2,
no panics, hangs, crashes or OOM. Newly ok: metadata-cache-image files,
the CVE open-time checks, v1 chunk-key lookup as libhdf5 does it, the
scale-offset fixes, Blosc2, and harness corrections. Baseline raised.
h5rs check --data flags none of the 435 fully-read ok files.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
cve-2025-44905 now reads as h5py reads it (the v1 chunk B-tree lookup),
leaving 5 our-errors: cve-2025-2308, cve-2025-44904 and
bad_nbit_parms_walk (corrupt data HDF5 2.0 reads through a bug), and
the Blosc2 and ZFP filters. The five unloadable-cache-image files stay
ok, now with the library behaving as the probe reports.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
clippy (with the async feature) flags chunks_exact with a constant size
in the AsyncHDF5File cache-image test added in d493d47.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The scale-offset fix (d110b1d) was covered only by unit vectors from
CVE chunks, and documented as three corner cases. The review found it
is much bigger: of 1480 scale-offset datasets h5py writes (every
integer type i1..u8, f4 and f8, both byte orders, with and without a
fill value, scaleoffset 0..full width), v2.7.0's decoder read 332
differently from h5py: 151 returned wrong values with no error (82
integer datasets with scaleoffset=0 and a wide range, 51 full-width
i4/u4/i8/u8, 18 f4 D-scale datasets with a large range) and 181 failed
to read. The cause in every case is a chunk libhdf5 stores at full
width, whose elements were decoded as offsets from minval.
tests/scaleoffset_interop.rs generates that matrix with h5py at test
time, stores h5py's decoded values uncompressed next to it, and
compares every dataset's bytes. It passes on this branch; with the
filters.rs before d110b1d it reports "332 of 1480 scale-offset datasets
differ from h5py".
CHANGELOG: a Correctness entry stating this was silent wrong data in
every release that decoded scale-offset (v2.2.0 to v2.7.0), replacing
the corner-case wording. docs/known-issues.md: a fixed entry with the
affected cases.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
File, MmapFile and LazyFile decoded the superblock extension and laid a
metadata cache image over the file's metadata; the other readers did
not, so the same file read differently by entry point: NativeVol,
AsyncHDF5File and MpiVol (clawhdf5-io) and the external source files of
a virtual dataset (clawhdf5-format vds.rs) read a file with an image
from its own bytes, which libhdf5 does not (they may be stale, or zeros:
h5clear_mdc_image.h5 failed with InvalidObjectHeaderVersion(0)), and
skipped the extension checks File::open makes (cve-2020-10810/10812).
Each of them owns its buffer, so each now calls the shared
superblock_ext::apply_cache_image_in_place, which checks the extension
and writes the image's entries in place (only the image block is
copied). These readers read whole datasets and cannot open a file and
fail each object, so an image libhdf5 cannot load is refused with the
image's error, never read around. clawhdf5-io's vol::load_hdf5 wraps it
for NativeVol (at open; for from_bytes the error is reported on read,
as a truncated file already was) and MpiVol. The MpiVol edit is minimal
and was not compiled: the mpi-io feature needs an MPI installation this
machine does not have (mpi-sys's build script panics).
Tests: NativeVol (open_path and from_bytes), AsyncHDF5File and a VDS
whose source file is h5clear_mdc_image.h5 (vds_interop.rs, against
h5py) read the fixture's values; the corrupted-image variants are
refused. Each fails without its fix.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
For a metadata cache image libhdf5 cannot load, libhdf5 opens the file
and fails the first metadata read (the image loads on the first
H5C_protect after open); h5py reports the error on the root group. The
conformance probe reported it that way, but File::open refused the file,
so the gate counted cve-2025-6269-1..4 and cve-2025-6516 as agreeing
with h5py for behaviour the library did not have.
The library now behaves as the probe reports: File (mmap, buffered and
from_bytes) and MmapFile open the file and every object lookup (dataset,
dataset_at, group, group listings and attributes, VL decoding) fails with
the image's error; LazyFile reads the root group's header at open, so
its open is that first read and fails. Probe and library take the
three-way decision (refuse at open / image loads / image cannot load)
from the same clawhdf5_format::superblock_ext::cache_image_state.
One deliberate difference from libhdf5 remains, documented: after the
failed first read libhdf5 reads the file's own metadata, which the image
was meant to replace and may be stale; here every lookup keeps failing.
File::cache_image_error / MmapFile::cache_image_error expose the error
to code that parses as_bytes() itself; h5rs checks it before reading any
object header (h5rs ls on cve-2025-6269-1 said "invalid object header
version: 0" from the stale bytes).
Test: metadata_cache_image.rs an_image_libhdf5_cannot_load_fails_every_object
(the fixture with its image signature broken; h5py opens that file and
fails the first read with "Bad metadata cache image header signature").
It fails on the previous commit, where File::open refuses the file.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
apply_cache_image returned a copy of the whole file with the image's
entries written in, and File (mmap by default), MmapFile and LazyFile
used that copy for every read: opening a 1 GiB sparse file with an image
needed 2 GB of memory, and an 8 GiB one aborted the process, where
de2a53f (which ignored the image) opened them in a few MB.
The metadata parsers read one contiguous slice, so the image still has
to be laid over the file's bytes; it is now laid over a private copy
that costs only the pages it touches:
- clawhdf5_format::superblock_ext::CacheImage decodes the image into an
entry list (address, offset in the block, length) and applies it to
any destination; cache_image_state tells an opener whether the file
has no image, a loadable one, or one libhdf5 cannot load;
apply_cache_image_in_place is for readers that own their buffer.
apply_cache_image and metadata_view (which copied) are gone.
- clawhdf5_io::HDF5Read::private_copy returns a writable private copy
of a reader's bytes: MmapReader gives a MAP_PRIVATE copy-on-write
mapping (memmap2 map_copy), so only the pages the entries land on are
copied; the default copies the bytes (in-memory readers).
- File, MmapFile and LazyFile write the image into that mapping
(crate::cache_image). File::from_bytes / open_buffered patch their own
buffer in place, copying only the image block, as libhdf5 does. A
file without an image is read straight from the mapping, unchanged.
An image entry that runs past the end of file is now refused: libhdf5
checks only that it starts inside the file, and the images libhdf5
writes never do this, but those bytes have nowhere to go in a view of
the file.
Tests: tests/cache_image_memory.rs has libhdf5 (through ctypes) add an
image to a 1 GiB sparse file and bounds resident-memory growth for all
three openers at 256 MiB; it fails on the previous commit (File::open
grew 2,148,720,640 bytes). reader.rs zero_copy_tests check that a file
without an image is read from the mapping itself and that an image goes
into a copy-on-write mapping, not a heap copy; clawhdf5-io checks that
private_copy writes never reach the reader or the file.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The review suggested libhdf5 loads every image entry and resolves
flush-dependency parents afterwards. It does not:
H5C__reconstruct_cache_contents (HDF5 1.14.6 and 2.0.0, and develop)
inserts each entry and then searches the cache index for its parents in
the same loop, failing with "fd parent not in cache?!?" when one is
missing. So a parent must be an earlier image entry, as before, or
metadata cached before the image loads: the superblock (address 0) and
the superblock extension's object header, which libhdf5 reads to find
the image. Those two were refused as parents; they are now accepted.
A parent listed after its child is still refused, as libhdf5 refuses
it, and so is an entry that is its own parent ("Child entry flush
dependency parent can't be itself").
apply_cache_image takes the superblock to know the extension address.
Test: superblock_ext::tests::flush_dependency_parents_must_already_be_cached
(parent-first loads, child-first refused, extension header accepted,
self-parent refused); the extension-header case fails without the fix.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
libhdf5 does not walk the chunk B-tree to read a dataset: it looks each
chunk up (H5B_find with H5D__btree_cmp3 and H5D__btree_found), asking for
the element-size coordinate as 0. collect_chunk_info_checked now parses
the tree with its keys and returns each stored chunk only when that
lookup, replayed over the scaled keys, finds it.
A key with a non-zero element-size coordinate is therefore found in a
1-D dataset (cmp3 compares only the first coordinate there, and found
compares with <=) and missed in a dataset of rank 2 or more, which reads
fill values. The previous commit refused every such key, which refused
1-D files libhdf5 reads correctly; before that, the rank-2 case read the
chunk's data where h5py reads fill values (cve-2025-44905
/Shuffle_float_data_le, now identical to h5py, so it leaves the
conformance report's list of libhdf5 bugs).
Test: chunk_keys_with_an_element_offset_read_as_libhdf5_reads_them
compares 1-D and 2-D files against h5py's values. It fails on the
previous commit (the 1-D file is refused) and with the refusal removed
(the 2-D file reads 0..23 where h5py reads fill values).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The Blosc2 fuzz tests checked only for panics and the size of the output,
so the offsets-chunk amplification passed all 40,000 iterations.
tests/blosc2_alloc_bounds.rs now measures peak allocation (a counting
global allocator) and asserts it stays within 6x the HDF5 chunk size plus
twice the input plus 2 MiB (ruzstd's fixed state) for:
- 20,000 mutated fixture frames, decoded with their real chunk size as
the limit, with edits aimed at the frame's and chunks' size fields;
- 20,000 mutated first chunks on their own;
- 5,000 frames built from random header sizes, offsets chunks and B2ND
shapes (padding chunk and block shapes, chunks larger than the array,
NaN, zero and repeated-value chunks).
Against the code before this series every test in the file fails (the
fuzz tests at frame iteration 3913, a zstd window, and random frame 289,
the offsets chunk); now the worst frame peaks at 0.48 of the bound.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
ruzstd reserves a frame's declared window (up to its 100 MiB default)
when a decoder is reset for a new frame, before decoding anything. The
Blosc, Blosc2 and bitshuffle decoders reuse one decoder per chunk, so a
Blosc2 chunk of two 16-byte streams, each declaring a 96 MiB window,
allocated 128 MiB. zstd_decode_into now sets the decoder's maximum window
to twice the stream's output (at least 128 KiB): c-blosc, c-blosc2 and
bitshuffle compress each block in one call with its size known, so
libzstd's window never exceeds the block.
Found by tracking peak allocation in the Blosc2 fuzz test.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
B2ND chunks were decoded whole, padding included, with up to 16x the HDF5
chunk size as their limit, so a crafted frame made each chunk allocate and
fill up to 16x the output (4 GiB for a 256 MiB HDF5 chunk). The padding is
the real bound (prod(ceil(c/b)*b) per chunk), but that can be 2^ndim times
the array, so it is no longer held at all:
- A Blosc2 chunk is now decoded block by block (decode_blocks), each block
handed to a sink as it is ready, with at most three blocks of scratch.
blosc2_decompress_chunk and plain frames still collect every block.
- reassemble places each B2ND block straight into the output and skips
blocks that are all padding (they are not decoded unless the delta
filter needs the first block). A frame's NaN chunks are handed over one
B2ND block at a time and its zero chunks cost nothing.
- A B2ND chunk must decode to exactly its padded size, its Blosc2 blocks
must be whole B2ND blocks no larger than the output, and a chunk may not
be larger than the array (hdf5-blosc2's chunk is the array), so a block
is never larger than the output.
Peak allocation for a 10-D array padded to 13x (NaN, repeated-value and
stored-block chunks) and for a 16x chunk was 4.5 MB and 17.8 MB for
315 KB and 1 MiB outputs before, and is now within the tests' bound.
Blosc2 files written by hdf5plugin in 9-D and 12-D, an 8 MiB single chunk,
1x1x1 and edge-chunk shapes still read exactly.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
A chunk header with nbytes 0 and no special type kept its declared block
size (up to 512 MiB): the block size was clamped to nbytes only when nbytes
was positive, and the scratch blocks were allocated before the (empty)
block loop, so a 20-byte chunk allocated about 1 GiB. The block size is now
clamped to nbytes always, and an empty chunk returns before any scratch is
allocated.
A frame chunk must also decode to the size the frame header gives it
(chunksize, or the remainder for the last chunk), and is decoded with that
as its limit, so an empty chunk in a frame for a non-empty HDF5 chunk is an
error rather than an empty result.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
parse_frame sized the offsets chunk from the frame header's own nbytes and
chunksize, so a 173-byte frame declaring 32 Mi chunks, with a 40-byte
repeated-value offsets chunk, built 256 MiB (up to 2 GiB) of offsets for a
1 MiB HDF5 chunk and then returned 4 bytes. The offsets chunk is now capped
at the output limit (at least 128 bytes); a frame whose nbytes/chunksize
imply more chunks than that is refused before anything is allocated.
tests/blosc2_alloc_bounds.rs measures peak allocation with a counting
global allocator; the reviewer's frame failed it (decoded Ok(4)) before.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Review follow-ups. With run_with_helpers a one-thread rayon pool gave each
read a second core (the caller plus the worker), so --decode-threads 1 no
longer matched h5py's one core per call; such a pool now adds no helper.
Per-thread decode scratch is kept up to 1 MiB per buffer (was 4 MiB),
bounding what never-exiting pool workers hold.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Conformance on tank, conformance/run.sh --no-fetch (2026-09-26): 597 of
697 ok, 6 our-errors (4 corrupt objects HDF5 2.0 reads through a bug, the
Blosc2 and ZFP filters), 2 mismatches (the known h5py big-endian VL bug).
Closes the known-issues entries for metadata cache images,
cve-2024-32624, cve-2020-10810/10812, and unfiltered chunks of the wrong
size; the N-Bit / 64-bit scale-offset entry is recorded as not our bug.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Four of the remaining our-errors are objects the reference (h5py 3.16 /
HDF5 2.0) reads only through a libhdf5 bug, and clawhdf5 refuses:
cve-2025-2308 (scale-offset codes past the end of the chunk),
cve-2025-44904 (short unfiltered chunks), bad_nbit_parms_walk.h5 (an N-Bit
parameter list one value short; libhdf5's own test_filter_bad_params now
requires the read to fail) and cve-2025-44905 /Shuffle_float_data_le (a
chunk key libhdf5's lookup misses, reading fill values). report.py lists
them under Known not-our-bug and counts them in the summary; they stay
our-errors in the class counts.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
libhdf5 loads a metadata cache image when it first reads metadata (the
root group), not at open, so for cve-2025-6269-1..4 and cve-2025-6516 (all
corrupt images) h5py opens the file and fails on "/". The probe reported
the image's error as an open error, which made those files our-errors;
it now records it on the root object, where h5py reports it. File::open
still refuses such a file outright: nothing in it can be read.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Placing a chunk cast its u64 offsets to usize; on a 32-bit target an
offset past the address space wrapped into the output (and could then
overlap another chunk's region when chunks are placed concurrently).
Such an offset is past the dataset, so it now saturates and the chunk
writes nothing, as the concurrent-placement check already assumed. No
change on 64-bit targets, where the cast cannot wrap (so no test here).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
CHANGELOG entry for the chunked read changes, and the known-issues entry
on concurrent chunked reads updated: both causes it names (per-read page
faults, readers waiting on a small pool) are fixed; the 16-thread
comparison with h5py stays open until re-measured on an idle machine.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
- A dataset without filters stores every chunk at the chunk's full size.
A chunk its index records at another size was read at that size, with
the rest of the chunk left as zeros (cve-2025-44904's
Scale_offset_float_data_le: 38- and 37-byte chunks for 48-byte chunks,
where HDF5 2.0 fills the rest with whatever its buffer held). It is now
refused, as later libhdf5 releases refuse it ("incorrect chunk size
returned from index for unfiltered chunk"):
chunked_read::list_chunks_for_read, used by every read path.
- A v1 B-tree chunk key carries 0 in the element-size dimension. libhdf5
compares that coordinate when it looks a chunk up, so whether it finds a
chunk keyed otherwise depends on where the key falls (in cve-2025-44905
/Shuffle_float_data_le, offset 4096, it does not, and h5py reads fill
values); we read the chunk. Such a key is now refused.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Scale-offset (filter 6) now follows H5Z__filter_scaleoffset:
- the packed codes start at byte 21 whatever size the chunk records for
minval (libhdf5 reads min(8, size) bytes of minval and always starts the
codes at buf_offset 21). We started them after minval plus 8 bytes, so a
chunk recording a size of 0 (cve-2025-44905 /Scale_offset_short_data_be)
decoded differently from h5py;
- with a fill value defined, a code equal to the all-ones code of minbits
bits is the fill value, including minbits 0 (code 0): a chunk of nothing
but fill values read as minval;
- minbits of the full width stores the elements as they are (no minval
added), and an integer scale factor of the full width means the chunk was
left untouched; minbits or a scale factor wider than the type is an
error;
- the class parameter (integer or float) decides the decode, a scale type
that does not match it is refused, and E-scale is refused, as in libhdf5
(no library writes it; it was decoded here unchecked);
- minval is the stored bytes zero-extended, as libhdf5 reads it.
Codes past the end of the chunk stay an error, as in libhdf5 releases
after 2.0 ("Buffer too short"; 2.0 reads past the buffer, cve-2025-2308).
Shuffle (filter 2) uses its own parameter as the element size, as libhdf5
does, instead of the dataset's element size; a parameter larger than the
chunk leaves the chunk as it is (cve-2025-44905 /Shuffle_float_data_be),
and a parameter of 0 is an error.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
libhdf5 decodes the messages of a v2/v3 superblock's extension when it
opens a file (H5F__super_read) and refuses the file when one does not
decode. We never looked at them, so we opened cve-2020-10810 (a File
Space Info message too short for the free-space manager addresses it
announces) and cve-2020-10812 (a metadata cache image past the end of the
file), both of which libhdf5 refuses.
A file written with a metadata cache image keeps its metadata cache
entries in an image block the extension points at; libhdf5 loads them
over the file's own bytes before it reads any metadata
(H5C__load_cache_image, H5C__reconstruct_cache_contents). In
h5clear_mdc_image.h5 the root group's header exists only in the image, so
every reader failed with InvalidObjectHeaderVersion(0).
The new clawhdf5_format::superblock_ext module:
- read_superblock_extension decodes the v1 B-tree K, File Space Info and
Metadata Cache Image messages with libhdf5's checks (versions, page size
512 B .. 1 GiB, the addresses a persisting message lists, the image
inside the file), with the new FormatError::InvalidSuperblockExtension;
- apply_cache_image checks an image block as libhdf5 does (signature,
version, recorded length, entry types, rings, ages, addresses inside
the file and not repeated, flush-dependency parents) and returns the
file's bytes with every entry written at its address
(FormatError::InvalidCacheImage);
- metadata_view does both.
File, MmapFile and LazyFile (and so h5rs) call metadata_view at open and
read an image file through the patched copy; the conformance probe does
the same. The image's trailing checksum is not verified, as libhdf5 does
not verify it. tests/fixtures/h5clear_mdc_image.h5 is libhdf5's own test
file.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Record the `blosc2` feature in the changelog, the README's feature table
and the crate table, and mark the Blosc2 half of the known "Filters"
issue fixed (dated, with the conformance run that shows h5ex_d_blosc2
reading). What stays open: ZFP, writing Blosc2, and the Blosc2 features
hdf5plugin never writes (dictionaries, lazy chunks, variable-length
blocks, user-defined codecs and registered filters), which are errors.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
hdf5plugin's Blosc2 was a clear "not implemented" error. It stores each
HDF5 chunk as a Blosc2 contiguous frame; for chunks of 2+ dimensions the
frame holds a B2ND array whose data are cut into (padded) blocks stored
one after another.
filters_blosc2 (feature `blosc2`, in `plugin-filters`; the facade
forwards both) decodes, following c-blosc2's decoder and hdf5-blosc2's
blosc2_filter.c:
- the frame: the msgpack header's fixed fields, the metalayer index, the
offsets chunk (with the special zero/NaN/uninitialised offsets) and
chunk lookup;
- Blosc2 chunks: 16- and 32-byte headers, special chunks (zeros, NaN,
uninitialised, one repeated value), split and unsplit streams, zero and
run-length streams, and the filter pipeline run backwards (shuffle,
shuffle with a byte-group size, bit shuffle including the version-2 and
later handling of a partial group of 8, delta against the first block,
truncated precision);
- the codecs, shared with Blosc 1: BloscLZ, LZ4/LZ4HC, Zlib, Zstandard;
- B2ND arrays: blocks gathered into C order, padding dropped, several
chunks per array, and the array shape checked against the chunk shape
in cd_values as the HDF5 filter does.
Dictionaries, lazy chunks, variable-length blocks, user-defined codecs
and registered filters (e.g. bytedelta) are errors. Uninitialised chunks
read as zeros. No encoder.
Tests: h5py + hdf5plugin write every codec x filter (none, shuffle,
bitshuffle, delta) and levels 0-9 over the plugin-filter cases, then
i1..u8/f4/f8 in 1-D to 5-D chunks with partial edge chunks, datasets of
zeros, one value and NaN, and Fletcher32 before Blosc2 (plain frames for
n-D chunks); clawhdf5 reads each exactly as its unfiltered twin, and
truncated precision exactly as h5py reads it. Fixture frames from
python-blosc2 (tests/fixtures/blosc2/generate.py) cover what hdf5plugin
never writes: special chunks, delta over many blocks and odd type sizes,
odd bit-shuffle blocks, forced splitting, multi-chunk B2ND arrays with a
zero chunk, and the refused features. The decoder is fuzzed (random and
mutated frames and chunks: no panic, output within the limit).
Conformance: h5ex_d_blosc2.h5 now reads (576 of 697 ok, baseline 575).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
libhdf5 checks a dataset's storage when it opens the dataset
(H5D__contig_check, H5D__compact_init): the element count times the
element size must not overflow, contiguous storage must end inside the
file, compact data must be the dataset's size. File::dataset opened
cve-2024-32624's /Dset_OBJREF (2^62 + 2 references of 8 bytes) and
reported its shape; only reading failed.
data_read::check_dataset_storage makes those checks (new
FormatError::InvalidDatasetStorage), and File, MmapFile and LazyFile run
it whenever they open a dataset (by path, by address, from a group), as
does the conformance probe. As before, a datatype, dataspace or layout
that does not decode is left for the read to report, so such a dataset
still opens and its attributes still read. An empty contiguous dataset at
a defined address, which libhdf5 refuses, is still accepted: clawhdf5 up
to v2.7.0 wrote them.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
A selection read (a hyperslab or points covering at most half the
dataset) decoded each chunk it overlaps into fresh buffers, one per
filter stage; it now uses the thread's chunk-decoding scratch like the
full readers. Covered by tests/chunked_read_paths_interop.rs (small
hyperslabs and points over every filter and type) and the partial-read
equivalence tests.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
read_f32/read_f64/read_i32/read_i64/read_u64 of a chunked dataset that
stores exactly that type in native byte order now decode every chunk
straight into the Vec<T> they return (data_read::read_chunked_native),
on File (through its chunk cache), MmapFile and LazyFile. Before, the
chunks went into a byte buffer that read_as_* then copied into a second,
typed one: two dataset-sized allocations and a full extra copy per read.
The output is zeroed pages from the allocator, backed by transparent
huge pages when large, like the byte reader's. Other types and byte
orders, and datasets with no storage or external data, keep converting
through the byte readers; unallocated chunks read as the fill value as
before.
tests/chunked_read_paths_interop.rs checks every chunked read path
(File twice, so cached; from_bytes; MmapFile; LazyFile; small, strided
and point selections; with and without the parallel feature) against
h5py for 1-8 byte integers and 2-8 byte floats in both byte orders,
through deflate, shuffle, Fletcher32, LZF, SZIP and Blosc, with partial
edge chunks, sparse datasets with default and non-default fill values,
and datasets larger than the chunk cache. A filter this build lacks must
be an error (or, when an optional filter declined every chunk, the right
data).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
h5py's track_order=True orders attributes as well as links; the writer
tracked links only. A tracking object's header now sets the attribute
creation order tracked/indexed flags and carries per-message creation
orders, an Attribute Info message holds the next order (inline too),
and dense storage gets a type-9 creation-order index. The file default
applies to datasets, with DatasetBuilder::track_order per dataset; more
than 65 535 attributes on a tracking object is an error (libhdf5's
counter is 2 bytes). The reader lists such attributes in creation
order.
h5py lists them in order (inline, dense, 20 000 on one dataset) and
keeps numbering in r+ mode, including its inline-to-dense move.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
- A simple dataspace of rank 0 holds one element in libhdf5 (the product
of no dimensions; h5py reads it as shape ()). num_elements() said 0, so
cve-2020-18494's /dset1 failed with DataSizeMismatch { expected: 0 }.
- A contiguous dataset whose storage is larger than its elements reads:
libhdf5 reads the elements from the start of the storage and ignores the
rest (H5D__contig_check checks only that they fit in the file). We
required the sizes to be equal, so the scalar /Dset1 of cve-2024-32623
and cve-2025-2309 (240 bytes of storage for one int) failed. Storage too
small for the elements is still an error. data_read::contiguous_read_len
is the rule, used by every contiguous read path.
- Dataspace::parse refuses what H5O__sdspace_decode refuses: more than 32
dimensions, a rank on a scalar or null dataspace, a dimension larger
than its maximum (new FormatError::InvalidDataspace).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Inventory of the 109 whole-file parser functions, a measurement of the
metadata ranges clawhdf5 touches on three corpus files against libhdf5,
options (storage trait, virtual slice, metadata prefetch, userfaultfd)
with how ros3, h5py+fsspec, pyfive, h5wasm and object_store do it, and
an incremental plan: M0 name-index lookups, M1 metadata over a Storage
trait, M2 raw data with batched ranges, M3 object_store backend,
M4 wasm fetch-driven lazy loading.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
inventory.py counts the functions and call sites that take the whole
file as &[u8]; range-trace (standalone crate, x86-64 Linux) records every
load clawhdf5 makes from a file by mprotect + single-step, unchanged
library code; libhdf5_reads.py counts libhdf5's reads through h5py's
fileobj driver and prints a dataset's chunk extents.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
libhdf5 refuses to list a symbol-table group that has an entry with an
empty name (H5G__ent_to_link: "invalid link name"), so h5py cannot list
cve-2021-46244's /BAG_root. We listed it, with an object at "/BAG_root/"
(the empty name, pointing at address 0). resolve_v1_group_entries — the
listing — now fails with the new FormatError::InvalidLinkName; path
lookups still find the group's other names, as libhdf5's by-name lookup
does.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
libhdf5 decides what an object header is in a fixed order
(H5O__obj_class_real): a group if it has a Symbol Table or Link Info
message, a dataset if it has a Datatype *and* a Dataspace message, a named
datatype if it has a Datatype message. The conformance probe called any
header with a Data Layout message a dataset, so cve-2024-33874's /Dset1 (a
datatype and a layout, no dataspace), which h5py opens as a named
datatype, was reported as a dataset we failed to read
(MissingMessage(Dataspace)). The probe now classifies with
object_class().
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Full reads of a chunked dataset (the cached reader behind the facade's
read_* and the uncached one behind mmap/lazy files and verify_provenance)
now decode each chunk into this thread's reusable scratch buffers and copy
it straight to its place in the output. Before, the cached reader decoded
batches of 128 chunks into fresh Vecs and the uncached one decoded every
chunk of the dataset into its own buffer before assembling any: a new
256 KiB allocation (and its page faults) per chunk and per filter stage.
Chunks are still inserted into the file's chunk cache when the whole
dataset fits in it.
With the parallel feature the calling thread now decodes too, sharing the
chunks with whichever rayon workers are free (run_with_helpers): a helper
the busy pool only starts after the read is done returns at once. Before,
the caller handed every chunk to the pool and slept, so readers outside a
small pool (2-4 threads) queued behind its workers; with a one-thread pool
the reads went sequential. Chunks are placed concurrently only when the
index puts them on the chunk grid at distinct places (a corrupt index is
read one chunk at a time), and the error returned is still the first
failing chunk's.
Fix: a chunk stored unfiltered in a filtered dataset (every filter-mask
bit set) that is shorter than a chunk read as zeros where its data was
missing through the cached reader (the facade's read_*); it is now an
error naming the chunk, as the uncached reader already made it.
Regression tests, both failing before this change:
tests/busy_decode_pool.rs (both workers of a two-thread pool busy, four
readers) and short_unfiltered_chunk_of_a_filtered_dataset_is_an_error.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
ref.py and compare.py reported 13 files as mismatches or our-errors that
were artefacts of the harness, not differences between the readers:
- User-defined links (tall.h5, tudlink.h5, twithub*.h5, tmany.h5, ...):
h5py's `get(name, getlink=True)` reports a user-defined link as a
HardLink, so ref.py listed it as an object. Read the link type from
H5Lget_info instead.
- Objects h5py cannot open (cve-2019-8397/8398, cve-2021-46243,
cve-2024-32618): the probe deduplicates by header address, ref.py by
ObjectID, which an unopenable object does not have, so each extra hard
link to it was listed again. Deduplicate those by link address.
- Nested array types (tarray3.h5): h5py expands them into trailing dims;
hash_values stripped one level and numpy broadcast every element into a
whole subarray. Strip every level.
compare.py no longer compares the attributes or links of an object h5py
could not open at all (cve-2018-17438/17439, cve-2019-9151): h5py read
none, so ours are neither extra nor errors against it.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Dense link and attribute indexes and the chunk index for several
unlimited dimensions were single leaves, capping them at 65 535
records. btree_v2_write builds trees of any depth, with node capacities
and pointer widths from libhdf5's H5B2__hdr_init arithmetic (now shared
with the reader as btree_v2::node_info) and libhdf5's node sizes (512
dense, 2048 chunks). Indexes that fit the old one-leaf layout are
written byte for byte as before (compared for 10..65 535 links, attrs
and chunks, tracked and filtered).
Tests: 100 000 links (short names; long names with creation order),
70 000 attributes, 200 000 chunks (and 80 000 deflated), read by h5py,
h5dump and clawhdf5 and edited by h5py r+; h5rs check on the same
shapes, asserting depths 2-3.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
decompress_chunk_exact_with decodes a chunk into a DecodeScratch the
caller keeps between chunks, instead of a new Vec per chunk and per filter
stage. Deflate inflates into a kept buffer with a reset (not rebuilt)
inflater, shuffle interleaves into the other buffer, and Fletcher32 checks
and drops its checksum in place (on the stored bytes when it is the first
filter undone). Other filters go through the registry as before. Output
and errors are those of decompress_chunk_exact; a unit test checks that
for every pipeline shape and filter mask with one reused scratch.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
libhdf5 compares the name when two hashes are equal; the writer broke
ties by insertion order, and libhdf5 could not find one of two names
whose lookup3 hashes collide (k69209 / k155448). Test fails before the
fix with h5py's KeyError.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Idle tank at 408f69e, h5py re-run in the same session. Contiguous reads
went from 0.25x to 1.44x h5py (full) and 0.12x to 6.3x (256x256
hyperslabs) on one thread; deflate full reads at 8 threads 887 -> 2943
MB/s (h5py processes 3042). Full chunked reads at 16 threads are still
0.69x-0.76x h5py processes; the issue stays open.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
libhdf5 fails to read a VL element whose global heap address is
undefined (all 0xff), even at length 0 ("addr undefined"); we returned
"" (or an empty sequence) in every reader. Checked with h5py first:
libhdf5 writes a null element with address 0, which still reads as
empty, and h5py writes "" as a zero-size heap object at a real address,
so no file they write relies on the old behaviour. read_vl_bytes now
treats address 0 as null whatever the length, as VlResolver does.
Tests, each failing before: vl_data unit test (8- and 4-byte offsets,
lengths 0 and 1); clawhdf5 vl_data_interop
a_vl_element_at_the_undefined_heap_address_fails_like_h5py (also checks
where h5py writes ""); h5rs dump --json and check --data on the patched
`undef` dataset; clawhdf5-wasm vl_strings.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The READMEs said ds[...] reads only the selected elements, and the
facade's read_selection docs that only intersecting chunks are
decompressed. The bounding-box path runs only when the box covers at
most half the dataset; larger boxes (any strided slice across the
dataset), compact, virtual and unwritten datasets and chunked ones with
a non-default fill value decode the whole dataset. The READMEs, the
facade and format docs, the bindings' docstrings and known-issues now
say so, and how index lists are read.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
h5rs (dump, ls, diff, check --data) kept its own lenient VL decoder:
a heap object longer than its element was cut to the element's length
(libhdf5 and h5py refuse it), a null string printed "" where h5dump
prints NULL, the stored element size was trusted, and every heap
collection was kept as an owned copy for the whole run. It now resolves
each element with VlResolver::element / string_element (new: one element
in place, borrowing from the file), and refuses a VL type whose stored
element size is not 4 + offset size + 4, as File does. H5::heap_object
and its cache are gone. h5diff compares a null VL string equal to an
empty one; so does h5rs diff.
clawhdf5-wasm already resolved VL strings with read_vl_strings; it now
uses VlResolver and checks the stored element size before reading, as
File::read_string does.
Tests (h5py writes the files, patched for "a\0b", a null element and
mis-sized heap objects, with 8- and 4-byte offsets):
- h5rs_interop dump_prints_vl_data_like_h5dump: byte-identical to h5dump;
- dump_json_vl_values_match_h5py: h5py's values, errors where h5py fails;
- check_data_flags_mis_sized_vl_heap_objects;
- clawhdf5-wasm tests/vl_strings.rs: wasm, File and h5py agree.
All four fail before. check --data over the 150 cve_hdf5 CVE and fuzzer
files now passes 15 (h5dump rejects 8 of them), was 16 and 9: the
stored-size check flags cve-2024-32608. h5rs-check-ok-files.sh --data:
0 of 422 flagged; h5rs-fuzz.sh: clean on 180 files.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
test_threads_read_the_same_file passed with the GIL held. The new
test_reads_release_the_gil measures the longest stall of a spinning
Python thread while another reads: with py.detach removed from the read
it stalled 0.062 s of a 0.064 s read and failed; with it, about 3 ms.
test_errors_match_h5py now compares the result whenever h5py reads the
key, instead of only checking that we raise when h5py raises, over a
longer key list.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The changelog, known issues and README said a group holds up to 65 535
links while a group of about 17 000 was already unreadable. Record the
fixes (child indirect blocks, the next-block offset, the index leaf cap,
refusing oversized dense messages, hard-link memoisation, dataset
attribute overwrite) and the limits that remain true: 65 535 links or
dense attributes per object, and 65 515 bytes per dense message.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
ds[np.array(1)] went down the index-list path, where tolist() returns a
scalar and extracting a list of indices raised a confusing TypeError.
h5py treats it as an integer index; so do we now. The h5py comparison
keys include 0-d arrays (signed and unsigned) on each axis; they failed
before.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
b0a1e4f fixed this for group and root attributes only. Setting a dataset
attribute twice still wrote two attribute messages with one name, and h5py
read back the first value: set_attr("a", 1) then set_attr("a", 2) read as
1, and list(attrs) was ["a", "a"]. DatasetBuilder::set_attr now replaces
the earlier value, compact or dense. Likewise, a hand-set attribute named
like a provenance attribute (_provenance_sha256, ...) is replaced by the
computed one instead of being written next to it and read first.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Every ds[...] and g[k] resolved the path from the root again, two or
three times per open, and resolving a name in a large group scans its
links: visiting a group was O(n^2). 4000 scalar datasets in one group
took 39 s (v1 group) and 131 s (dense) to list, read and re-read; now
0.3 s each. A Dataset keeps its object address, a Group (and the file's
root) its address and, after the first lookup, its link table.
New facade API File::dataset_at(address), tested in integration_tests.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
A hard link's target may go through other hard links, and each was
resolved again every time a path went through it. With each link's
target naming the previous link twice (g/s{i} -> /g/s{i-1}/s{i-1}) the
work doubled per link: finish() took 46 s for 26 links in a debug build,
and 60 would never finish. Resolved links are now remembered, so the work
is linear in the links, and a hard link met again while it is being
resolved is reported as a cycle by name. The depth limit (64) still bounds
the recursion through links not yet resolved.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The link and attribute name indexes are one v2 B-tree leaf, sized to the
next power of two. libhdf5 takes a leaf's capacity from that node size,
but a leaf's record count is a 2-byte field. From about 47 700 links the
node had room for more than 65 535 records, so adding a link in h5py
overflowed the count: a group of 65 535 links crashed h5py, or could no
longer be listed ("unknown link class"). The node is now capped at a full
leaf of 65 535 records, so libhdf5 splits it instead.
Dense attributes now go through the same index builder. Their record
count was written modulo 65 536, without error; more than 65 535
attributes on one object are now refused, like links.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
A message in dense storage is a fractal heap object, and an object must
fit one direct block: 65 515 bytes here, since the writer has no
huge-object path. A bigger one (a soft link with a 80 000-byte target in
a group of more than 8 links) was written without error, cut off at the
end of its block, and libhdf5 could not list the group ("object overruns
end of direct block"). finish() now fails with an error that names the
limit, for links and for dense attributes; a 65 001-byte soft link target
still works and h5py reads it back. The heap packer also skips a child
indirect block whose blocks are all too small for the next object instead
of walking it.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Each run of consecutive indices was its own uncached hyperslab read, so
a list over a compressed chunked dataset decoded the same chunk once per
run (d[range(0, 200000, 40)] over 20 gzip chunks: 8 s, h5py 0.014 s).
Plan::reads now groups the indices — a group ends only where a whole
chunk holds no selected index, or, unchunked, at a gap over 64 KiB — and
the selected rows are gathered from each group's block in Rust. Now
3.8 ms (h5py 4.1 ms, release, tank). The new test (1-D, 2-D and
contiguous, compared with h5py, 2 s bound) took 5.8 s before.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Dense link and attribute storage keeps its messages in a fractal heap. Its
root indirect block holds direct blocks up to 64 KiB, 512 KiB in all; rows
past that are child indirect blocks. The writer kept adding rows of direct
blocks instead, and libhdf5 and h5rs read them as indirect blocks: a group
with 20 000 links of 20-byte names was written without error and could not
be listed ("incorrect metadata checksum"), and 150 dense attributes of up
to 56 KB could not be opened. The heap writer now follows the doubling
table: rows past the direct ones hold child indirect blocks, each with its
own rows, nested as deep as the heap needs.
Two more heap bugs are fixed on the way. An object bigger than the next
block's free space was written into it anyway and cut off; the block is
now left unallocated and the object goes in the first block big enough, as
libhdf5 skips blocks. And the header's next-block offset was 0, so libhdf5
adding a link to such a group overwrote the heap's first block ("bad
version number for message"); it is now the offset after the last block.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
VlResolver kept an owned copy of every object of every heap collection
it parsed, for the whole read. Collections nested inside each other's
object data, 32 bytes apart with each element pointing at a different
one, made retained memory O(elements x file size): 1.58 GB for a 744 KB
file (read_vl_strings did the same before VlResolver). Chaining every
collection's objects into one shared run of tiny objects made parse
time O(elements x objects) as well. libhdf5 refuses these files.
- The cache records where each object lies (GlobalHeapCollection::
parse_index, new) instead of copying it, and is dropped past a 32 MiB
budget.
- A collection overlapping one already read is an error: libhdf5 gives
every collection its own block, so only a crafted file has them.
- parse and parse_index refuse a collection that runs past the end of
the file and an object that runs past the end of its collection.
tests/vl_heap_bounds.rs measures peak heap use with a counting
allocator: 129 MB and 350 MB live before on its two crafted files (64 KB
and 176 KB), 97 KB and 0.9 MB now. Conformance unchanged at 575 of 697.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
np.concatenate copies structured dtypes field by field into np.empty, so
the padding of ds[[0, 3, 6]] held process memory. The runs' bytes are
joined in Rust, whole elements at a time, before anything becomes numpy:
the padding is the file's bytes (h5py's) and the result is still a view
of the Rust buffer. The h5py comparisons now compare every byte of
structured values; the new test failed on the padding before.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
PanicException derives from BaseException, so `except Exception` let a
library bug through. Every call from the bindings into the library now
runs under catch_unwind and a panic becomes InternalError (RuntimeError)
naming the object. Tests: a hidden hook panics inside the guard; and the
v4 chunk indexes are compared with h5py from Python — with the library
fix reverted, ds[0:30] of the implicit-index dataset now raises
InternalError instead of aborting the test run.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
read_raw_data_selection's chunked fallback (taken when partial_read
declines, e.g. a bounding box over half the dataset) handed the layout's
chunk dimensions, element-size dimension included, to
generate_implicit_chunks, which indexed past the dataset rank. It then
decoded the whole dataset regardless, so the enumeration is gone: the
arm decodes and extracts for every chunk index.
The new test reads small and large hyperslabs of all five v4 indexes
written by h5py and compares with h5py's values; it panicked before.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The review measured the default pool unchanged (about 2900 MB/s at 16
threads before and after) and still short of 16 h5py processes; small
pools still make outside readers wait. Say so instead of marking the
scaling issue fixed.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Full reads of chunked datasets handed their chunks to rayon. With a
one-thread pool (concurrent_read --decode-threads 1, RAYON_NUM_THREADS=1)
every thread reading through a File queued behind that single worker, so
16 readers decoded on one core: per-thread CPU time showed one thread
doing all the decoding and the readers almost none, and full reads
stopped at about 2x one thread. The cached full-read path and the
uncached reader behind verify_provenance now decode inline when the pool
cannot parallelise (parallel_read::pool_can_parallelise).
The File's chunk cache was the suspect but not the cause: datasets over
its budget were already read without inserting, and skipping its lookups
gained only a few percent at 16 threads.
The regression test keeps a one-thread global pool's worker busy and
requires a full read and verify_provenance to finish anyway; before the
fix both waited for the worker (timed out).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Setting a group or root attribute twice wrote two attribute messages with
the same name, and h5py read back the first value: set_attr("w", 1) then
set_attr("w", "two") read as 1. The later value now replaces the earlier
one, as `attrs[name] = v` does in h5py, including when a group is merged
from two builders.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The writer marked every link name ASCII, so a name such as "größe" was
stored as UTF-8 bytes under the ASCII character set (h5py reports cset 0
for it). Names that are not plain ASCII now carry the UTF-8 flag, as h5py
writes them; ASCII names are unchanged.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
FileWriter wrote the root group plus one level of groups, and refused
path-like names. The writer now flattens its builders into a group tree
(writer_tree.rs) before layout:
- A name may be a path ("a/b/x", "/a/b/x" at the root); missing
intermediate groups are created as h5py does, and GroupBuilder gains
create_group/add_group so builders nest to any depth. A group added at a
path that already holds a group is merged into it (require_group);
any other repeated name, an empty or "." component, or an absolute path
below the root is an error.
- add_soft_link, add_hard_link and add_external_link on FileWriter,
FileBuilder and GroupBuilder. Hard-link targets are resolved to objects
at finish (through other hard links; a missing target, a soft link on the
way or a cycle of paths is an error). Objects with several hard links get
an Object Reference Count message so libhdf5 can delete one link without
freeing the object.
- track_order(true) per group, or as the file default, tracks and indexes
link creation order: Link Info flags and max order, the order in each
Link message, and a type-6 creation-order B-tree for dense groups.
- A group's link index is one B-tree leaf; more than 65535 links is an
error.
Groups are laid out depth-first from the root, datasets group by group,
and untracked groups keep writing datasets, then groups, then other links:
files with one level of groups are byte-identical to before.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
clawhdf5-format now depends on libc on Linux (huge-page advice for read
buffers); the probe's committed lockfile picks that up.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
VL-string datasets (h5py's default str dtype) failed read_string with
"type mismatch: expected String, got VariableLength". read_string now
reads fixed- and variable-length strings, with h5py's values (a string
ends at a NUL, a null element is ""). New:
- Dataset::read_string_bytes: each VL string's exact bytes;
- Dataset::read_string_selection: hyperslabs/points of either kind;
- Dataset::read_vlen::<T>() and read_vlen_selection::<T>(): VL sequences
of numbers as Vec<Vec<T>>, T in f64/f32/i64/i32/u64, converted like the
other typed readers;
- File::decode_strings / decode_string_bytes / decode_vlen: VL values in
compound fields and AttrValue::Raw attributes;
- MmapDataset and LazyDataset: read_string for VL strings,
read_string_bytes and read_vlen.
tests/vl_data_interop.rs checks every path against h5py with 8- and
4-byte offsets: scalar, 1-D and 2-D, ASCII and UTF-8, empty strings,
contiguous, compact, chunked with gzip and shuffle, unwritten and partly
written chunks, hyperslabs, compound members, attributes, a big-endian
base type, and a patched file with an embedded NUL and mis-sized heap
objects. NetCDF-4 string variables read too (netCDF4-python test).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
README gains a Python section (maturin develop into a venv, a reading
example that was run against an h5py-written file, the supported types
and keys, what writing covers). The crate README says the same in more
detail. QUICKSTART showed clawhdf5.open()/read_f64(), which never
existed; it now shows File(...)[...].
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
A 256 x 256 hyperslab of a contiguous f32 dataset read at an eighth of
h5py's speed: partial_read copied the bounding box out of the file, the
extractor then walked it element by element (a recursive call and two
bounds checks per element) into a second buffer, and read_f32_selection
converted that into a third.
Selections of contiguous data are now copied straight from the file, one
memcpy per run of elements contiguous in the file (gather.rs: a block
along the last dimension, touching blocks as one range, whole rows
merged), with no zero-filled intermediate and no full copy for large
selections. The typed selection readers copy into their Vec<T> directly
when the dataset stores T natively (new data_read::read_selection_native
and sealed NativeElement trait, which the read_as_* fast paths now share;
read_as_u64 gains one) and convert as before otherwise. The general
extractor used by the chunked paths runs on the same run walker, keeping
its old handling of unvalidated selections.
Checked against h5py (contiguous_read_interop.rs) for strided, blocked,
adjacent-block and whole-row hyperslabs, points and empty selections of
every 1-8-byte type in both byte orders, ranks 1-4.
Also keeps the huge-page threshold constant out of no_std builds, where
it was unused.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
ci-test.sh gains a step that lints clawhdf5-py, builds its wheel with
maturin, unpacks it under target/ (the interpreter's environment is not
touched) and runs the pytest suite, which compares reads with h5py. It
skips without maturin/pytest, and fails instead under
CLAWHDF5_REQUIRE_INTEROP=1. The CI interop venv installs maturin and
pytest, so CI runs it.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Checked with h5py on a patched file:
- a VL string with an embedded NUL reads up to the NUL (libhdf5 converts
VL strings to C strings); read_vl_strings returned "a\0b";
- an element whose global heap object is not length x base size bytes is
an error ("Expected global heap object size does not match"); we
returned the object cut to the length;
- a heap address of 0 is a null element whatever its length.
vl_data::VlResolver does this, caching each parsed heap collection:
read_vl_strings parsed the whole collection again for every element.
read_vl_strings and read_vl_bytes use it; check_element_size refuses a VL
type whose stored element size is not 4 + offset size + 4. The
conformance probe resolves VL values through VlResolver instead of its
own lenient copy (575 of 697, unchanged).
The new unit tests fail against the old read_vl_strings.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
ds[key] read the whole dataset and sliced it in numpy, and knew six
dtypes. Keys (ints, positive-step slices, Ellipsis, one increasing index
list, compound field names) now map onto hyperslab selections, and the
facade's read_selection bytes become the numpy buffer without a copy
(PyArray::from_vec viewed as the dtype). dtype mapping follows h5py for
all integer/IEEE float widths and byte orders, bool, enum, complex, fixed
and variable-length strings, vlen sequences, opaque, array types and
(nested, padded) compounds; anything it cannot describe exactly is a
TypeError. Attributes return what h5py returns; groups and files gain
the rest of the h5py mapping interface. Reads run under py.detach.
tests/test_read_vs_h5py.py compares >500 reads with h5py 3.16 on an
h5py-written file, checks errors match, that a damaged chunk outside the
selection is never touched, and 8 threads reading at once.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
In a file with sizeof_addr = 4, a VL string attribute came back as
AttrValue::Raw, a compound's VL member failed with
GlobalHeapObjectNotFound and VL datasets failed with a size mismatch.
Two bugs: Datatype::type_size() said 16 for every VL type, while the
element is 4 + offset size + 4 bytes (12 here); and the global heap was
parsed without the padding libhdf5 puts after its collection and object
headers (both round up to 8), so with 4-byte lengths every object was
looked up 4 bytes early. Datatype::VariableLength now carries the size
its datatype message stores, and writes it back.
Checked against h5py in tests/vl_offset4_interop.rs (fails with either
fix reverted). Conformance unchanged at 575 of 697; in cve-2024-32608 a
VL attribute whose datatype claims 524304-byte elements is now an error
(h5py cannot iterate those attributes at all).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
A full read of a contiguous dataset is one memcpy from the mapped file,
yet ran at a quarter of h5py's speed on one thread: the fresh output Vec
took a page fault and a kernel page clear for every 4 KiB page written,
16384 per 64 MiB, costing several times the copy (the benchmark spent
6.2 s of 8 s in the kernel, 4.3M minor faults). numpy, so h5py, madvises
MADV_HUGEPAGE on allocations of 4 MiB or more; the typed readers' output,
the raw contiguous read and the chunk assembly buffer now do the same
(Linux only, libc as a Linux-only dependency; no-op otherwise).
New h5py comparison tests cover full and selection reads of contiguous
data for every 1-8-byte integer and float type, both byte orders, ranks
1-4, empty selections, and datasets past the 4 MiB threshold.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
pyproject.toml named the distribution rustyhdf5 while the extension
module is clawhdf5, and the package's tests imported rustyhdf5, so
pytest failed at collection. Distribution, module-name and tests now
agree; the module gains __version__. maturin develop + pytest: 28 pass.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
libhdf5 reads a group's Group Info message before it inserts a link, and
FileWriter wrote none, so h5py in "r+" mode could not add a link to any
group we wrote: "Unable to create link (message type not found)". Each
group header now carries a version 0 Group Info message with the default
link-phase thresholds, as libhdf5 writes for a new group.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
h5py threads stay flat (global lock); clawhdf5 hyperslab reads of deflate
data scale to 1244 MB/s at 16 threads (9.7x h5py threads, 0.89x h5py
processes). Two deficits recorded as open issues: full chunked reads stop
scaling at ~4 threads (chunk cache suspected), and contiguous reads are
4x (full) to 8x (hyperslab) slower than h5py single-threaded.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
With the header checks merged, the library refuses truncated files and
misaligned chunk-index keys itself, so check reported only "file cannot
be opened" for a truncated file. It now reports the truncation (stored
end of file vs file length) or the library's error, and the misaligned
chunk test accepts the library's refusal of the key.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Measured on tank on the 150 cvefiles/ and fuzzerfiles/ of cve_hdf5 (the
earlier text said 180): check --data passes 16 (was 28), and h5dump
1.14.6 rejects 9 of those (was 21). It still flags none of the 418
conformance files both readers read in full.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
parallel_integration.rs declared `_sequential` and used `sequential`
under the parallel feature, which nothing in CI enabled for the facade.
ci-test.sh now lints and tests the facade with parallel on.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The review read libhdf5's H5O__chunk_deserialize as accepting a v2
message that runs up to 4 bytes into the chunk's checksum, since it
bounds message bodies by the whole chunk buffer. It does not accept it:
the message loop stops at the checksum, and the checksum read that
follows starts past it and overruns the chunk ("ran off end of input
buffer while decoding"). h5py refuses such files whether the message
runs 1, 4 or 5 bytes in, and so does clawhdf5, with its own error text.
No code change; the test pins the agreement and a comment records why.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The truncated-file check and the end-of-file clamp reached File,
LazyFile and MmapFile but not clawhdf5-io's readers, which still opened
truncated files and read past the recorded end of file. NativeVol
(open, and read_dataset for from_bytes), AsyncHDF5File::from_bytes and
MpiVol's collective read now view the file through the new
vol::hdf5_view: from the superblock to Superblock::data_end, refusing a
file shorter than that.
MpiVol's read is compiled only with the mpi-io feature, which needs an
MPI installation; it was not built here. The edit there only swaps its
two-line superblock setup for hdf5_view.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
A chunked layout records the element size as its last dimension, and
libhdf5 refuses a dataset whose datatype has another size
(H5D__chunk_set_sizes: "stored datatype size in chunk layout does not
match datatype description"). clawhdf5 ignored the recorded size and
read the chunks anyway, for v3 and v4 layouts. The check runs on every
chunked read (read_chunked_data*, read_raw_data_selection) and compares
against the stored size: a variable-length element is 4 + offset size
+ 4 bytes, not Datatype::type_size's 16.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
libhdf5 encodes a version-4 layout's chunk dimensions in (log2(max) +
8) / 8 bytes, and HDF5 2.0.0 (h5py 3.16) refuses any other width:
"stored chunk dimension encoding length does not match value calculated
from chunk dimensions". The writer rounded 3 bytes up to 4, so h5py
could not open a dataset we wrote with a chunk dimension from 65 536 to
16 777 215, for every chunk index (single chunk, fixed and extensible
array, v2 B-tree). The three encoders now share push_v4_chunk_dims,
which writes the exact width.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
A version-4 layout stores every chunk dimension in the fewest bytes that
hold the largest one (H5D__chunk_set_sizes: (log2(dim) + 8) / 8), so a
chunk dimension of 65 536 to 16 777 215 takes 3 bytes. Only widths 1, 2,
4 and 8 were decoded; an h5py file with chunks=(70000,) and
libver='latest' failed with UnexpectedEof. Widths 1-8 are decoded now;
0 and more than 8 are refused with libhdf5's "encoded chunk dimension
size is too large", and a dimension past u32 is refused, not truncated.
The review asked for libhdf5's check that the stored width matches the
one computed from the dimensions. HDF5 2.0.0 (h5py 3.16) refuses any
mismatch, but HDFGroup/hdf5@e124c36 ("Allow reading of files with chunk
dimensions encoded using more bytes than necessary", 2026-06-05) relaxed
it to refusing only a width too small for the dimensions, which cannot
happen once the dimensions have been decoded from that width. Follow
current libhdf5: a wider-than-needed encoding is read. clawhdf5's own
writer produces such layouts (the next commit fixes that).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The short-decoding chunk (wrong data, pre-existing), the Blosc header
underflow (crash) and filter 32023 registration, each with its date and
what it changes; the conformance count is unchanged at 573 of 697.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The compound overlap check measured each earlier member with
Datatype::type_size, which is a fixed 16 for a variable-length type. On
disk a VL member takes 4 + offset size + 4 bytes, 12 in a file with
4-byte offsets, so a member right after one was refused as "member
overlaps with previous member" (and with the type, every attribute of
the object). libhdf5 measures members by their decoded, stored size
(times a v1 member's array dimensions); so does this now.
Reading VL values in such files is a separate, older gap, now recorded
in known-issues.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
With the pcodec feature, 32023 was a built-in entry (the legacy reader for
the pcodec chunks clawhdf5 <= 2.7.0 wrote under that ID), so
register_filter(32023, ...) was refused as "built in", although
UnsupportedFilter(32023) names Granular BitRound as not implemented and
the registry docs point to register_filter for such IDs.
That entry is now shared: it claims only chunks whose filter is named
"pcodec"; any other chunk with ID 32023 goes to the registered codec (or,
with none registered, gets UnsupportedFilter as before), and writing
32023 uses the registered codec. Every other built-in ID still refuses
registration.
Test: a_codec_can_be_registered_for_granular_bitround (registers, round-
trips chunks with no name and other names, still reads a legacy "pcodec"
chunk with the built-in reader, and after unregistering reads nothing).
It fails without the change ("filter 32023 ... is built in and cannot be
re-registered"). It and the existing legacy-pcodec test share a lock,
since the registry is process-wide.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
dump_matches_h5dump requires byte identity with the h5dump on PATH, and
CI's rust:latest (Debian 13.7) installs hdf5-tools 1.14.5, not the 1.14.6
the test was written against. Ran the whole clawhdf5-tools suite in
rust:latest with Debian's hdf5-tools and pip h5py 3.16.0 (HDF5 2.0.0), as
CI sets it up, with CLAWHDF5_REQUIRE_INTEROP=1: 18 of 18 pass, so the
comparison needs no loosening. The version is now printed in the CI log
so a future Debian update that changes the output is easy to spot.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
A null-padded fixed string inside a compound or an array member printed
trimmed ("" for three NULs, "a" for "a\0b"), where h5dump prints every
byte ("\000\000\000", "a\000b"); only top-level strings were shown in
full. DATA blocks now render elements through one function that keeps
the padding at any depth.
The README now lists the remaining known differences from h5dump:
nested compounds print inline, and long double values are printed as
errors (exit 1) with the datatype as an H5T_FLOAT block.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The README said check skips only "global heap collections other than
those a value read touches", but read_dataset returns the raw heap IDs,
so no collection was ever read: a file whose global heap collection
claims a 4 GiB object passed `check --data` with no problems, while
h5dump (and h5rs dump/diff) fail on it.
With --data, every variable-length element (strings and sequences, also
inside compounds, arrays and nested sequences) of every dataset and
attribute is followed into its collection. A collection that does not
parse, a missing heap object, or a sequence longer than its heap object
is a problem at the collection's address, once per object; the summary
counts the collections read.
Measured on tank, 2026-09-26: the 418 fully-read conformance ok files
still pass (scripts/h5rs-check-ok-files.sh --data, 0 flagged), and
`check --data` now flags 152 of the 180 CVE-corpus files (was 147); of
the 28 it passes, h5dump 1.14.6 rejects 21 (was 26 of 33).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
HDF5 stores every chunk at the full chunk size (edge chunks are padded
before filtering, and with "don't filter partial edge chunks" they are
stored raw at full size), so a filter pipeline that decodes to fewer bytes
means a corrupt chunk. Every chunk reader padded it with zeros and
returned it as data. libhdf5 returns the rest uninitialised, or fails when
the filter checks (Blosc with nbytes = 0).
New filters::decompress_chunk_exact decodes and then requires exactly the
chunk size, with the chunk's coordinates in the error
(ChunkedReadError "chunk at [16] decoded to 16 bytes, expected 32"). It
replaces decompress_chunk_masked at every chunk read path: the full read
(sequential and lane-partitioned), the cached read, the sweep read, the
planned-selection read, parallel_read's three decoders and partial_read's
box read. decompress_chunk_masked is unchanged (fractal-heap huge objects
already checked their own size). Blosc also rejects a frame declaring no
data where the chunk size is known.
Tests, each failing with the check disabled: filters and parallel_read
unit tests; h5py_short_decoded_chunk_is_an_error (gzip chunks rewritten
short with write_direct_chunk: 1-D, a 2-D edge chunk, and 40 chunks with
shuffle, read through File full/cached/selection reads, a selection that
avoids the chunk still reads, MmapFile and LazyFile, with and without the
parallel feature); plugin_filters_interop short_decoding_chunks_are_errors
(Blosc nbytes=0 and short, LZF and bzip2 short; the Blosc nbytes=0 case
read as 16 zeros before). The existing don't-filter-partial-edge-chunks
tests still pass. Conformance (tank, 2026-09-26): 573 of 697 ok, and no
file changed class, reader result or first issue against the pre-fix run.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The reader now refuses a compound with a repeated field name or no
fields and an enum member with an empty name, as libhdf5 does, but the
writer still wrote them: CompoundTypeBuilder and EnumTypeBuilder build
them without complaint, so clawhdf5 wrote files it could not read back.
They were never valid HDF5; h5py refuses them.
Datatype::check_encodable, which FileWriter::finish runs on every
dataset and attribute type, now parses the type's own encoding back and
refuses one the reader refuses, with the reader's reason. That keeps the
writer in step with every reader check, not only these three.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
-c meant "list at most N differences" in h5rs, but in h5diff -c is
--compare (a flag) and the count is -n/--count=N, so a script moved over
from h5diff behaved differently: `h5diff -r -c 2 A B` exits 2 (the 2 is
taken as a file name) while h5rs exited 1.
The count is now -n/--count, -c/--compare is accepted (h5rs always lists
objects that are not comparable), and the --count=N, --delta=D,
--relative=R forms are accepted; exit codes equal h5diff's on 7 cases.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
An OBJ that was a soft link was resolved and its target object compared,
so two files whose /g/s both point at /z differed when /z did: exit 1,
where h5diff (without --follow-symlinks) compares the links' target paths
and exits 0.
A soft link is now compared as a link wherever it is, OBJ included.
--follow-symlinks compares the objects soft links lead to instead, walks
into soft-linked groups, resolves relative targets against the link's
group, and treats two dangling links as the same; exit codes equal
h5diff's on 14 cases. External links are never followed (documented).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
libhdf5 refuses a chunk of 4 GiB or more only when a version-1 B-tree
indexes it (H5D__chunk_init: "chunk size must be < 4GB with v1 b-tree
index"). HDF5 2.0 writes larger chunks with layout version 5, and h5py
reads them; these were refused. chunk_geometry now takes the layout
version and applies the limit to layout version 3 and earlier only.
The interop test is ignored by default: h5py writes a 4 GiB chunk and
both libraries hold it in memory.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Audited LZF, bitshuffle, bzip2 and Blosc/BloscLZ for arithmetic on
header fields and unchecked slicing. The only live bug was the Blosc
frame-size underflow fixed in the previous commit; bzip2's output-growth
step now uses a saturating subtraction as well (the allocator may hand
back more capacity than asked for).
src/test_fuzz.rs (tests only) feeds each decoder random bytes, truncated
seeds and one-to-four-edit mutations of valid frames, biased towards
edge-case u32 values in size and offset fields, and asserts no panic and
no output over the limit (tests build with overflow checks and debug
assertions). Per decoder: LZF, bzip2, bitshuffle in all six mode/block
settings plus hostile cd_values, Blosc across four codecs, three shuffles,
stored frames and a hand-built BloscLZ frame, and BloscLZ streams alone.
With the previous commit's check removed, fuzzed_frames_never_panic panics
at the same subtraction. A 100x-iteration soak (different seed) found no
other panic.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
With a tolerance, integers were converted to f64 before comparing, so
int64/uint64 values above 2^53 that differ compared equal: -d 0 on 2^60
and 2^60 + 1 exited 0, where h5diff exits 1. Integer pairs are now
compared in i128 (the delta against floor(D), the relative quotient from
an exact difference), and the report prints the exact difference.
h5diff compares exactly when -p is below the f64 epsilon (2^60 and
2^60 + 1 differ at -p 1e-18, and nextafter(2, 0) and 2 at -p 1.5e-16);
h5rs now does the same.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The path walk skipped the second hard link to an object, so a file that
shares one dataset between /x and /y differed from a file holding two
identical copies: "</y> exists only in <B>", exit 1, where h5diff exits 0.
For a hard-linked group every member was reported the same way.
diff now enumerates every path below the start object (a hard link back
to an ancestor is recorded but not descended into), so each name is
compared. A group whose links cannot be read is now an error instead of
an empty group.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
A hostile chunk whose header gave a compressed size below 16 bytes, not
stored raw, made the block-table check subtract past zero: a panic in any
build with overflow checks (cargo test, maturin develop, debug CLI). The
frame size is now checked against the header size, and the stream-length
read no longer adds to an untrusted offset.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
CHANGELOG (Correctness): the new header, datatype, chunk and truncation
checks, what is left out on purpose (checks HDF5 2.0 lacks; the two
v2.7.0 writer quirks), the conformance numbers and the new FormatError
variants. known-issues: the "Header checks" audit gap is fixed, with the
one CVE object and two CVE files libhdf5 still refuses and we read.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
h5py writes a dataset (libver earliest, so version-1 object headers) and
the script damages one field of a copy: a layout message flagged
shareable, a message size that is not a multiple of 8, a compound field
that repeats an earlier name or overlaps it, an empty enum member name, a
float exponent overlapping the mantissa. h5py refuses every damaged copy,
and clawhdf5 must refuse exactly those and read the valid files. All of
them but the unaligned one (then an UnexpectedEof) read before this
branch. The helper now reads any datatype
(File::read_multi) rather than only integers.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Two of the datatype checks added on this branch refused files clawhdf5
itself wrote up to v2.7.0: it put the sign bit of every float at
position 63 (so every f32 it wrote failed "sign bit position out of
bounds", including every agent store's embeddings), and wrote an
empty-string attribute with a size-0 string type ("invalid datatype
size", failing every attribute of the object). libhdf5 refuses both, but
neither decodes to wrong values (an IEEE float's sign position is not
used; a size-0 string is empty), so this reader keeps accepting them.
New fixtures written by clawhdf5 v2.7.0 (FileBuilder with every datatype,
layout and attribute kind it could write, and a FileWriter paged file)
and legacy_writer_files.rs, which reads every object of them. The agent's
v2.5.0 store fixture (float16_store) passes again.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Measured on the CVE corpus: check --data passes 33 of 180 files, and
h5dump 1.14.6 rejects 26 of those. Recorded under the open "Header checks"
gap and in the crate README.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The superblock records where the file's data ends. libhdf5 refuses to
open a file shorter than that ("truncated file", H5F__super_read) and
fails any read past it ("addr overflow" / "address plus size exceeds
file eoa"). clawhdf5 read whatever was left of a truncated file, and read
bytes after the recorded end as if they belonged to the file.
New Superblock::data_end: FormatError::TruncatedFile for a file shorter
than its recorded end, otherwise where the HDF5 data ends. As in libhdf5,
the recorded end moves with the superblock when its recorded base address
is not where it is (a user block added afterwards; cve-2021-36977, which
h5py reads, depends on it), and the check is skipped for a v3 superblock
still being written in SWMR mode. File, LazyFile, MmapFile and the
conformance probe refuse a truncated file and parse only up to the end.
Files clawhdf5 writes record their true length, and the v2.5.0 agent-store
fixture and files written by v2.7.0 (plain, paged, user-block free) pass
the check.
Interop test: h5py writes a file; a copy missing its last 8 bytes must be
refused by both, a copy with bytes appended and one moved behind a new
512-byte user block must read in both.
Conformance (cached corpus, tank): 570 -> 571 ok
(h5clear_fsm_persist_less.h5, whose data past the recorded end was being
read); ten files h5py refuses as truncated (cve-2018-13874,
cve-2018-13876, the family/multi/subfiling members, h5clear_fsm_persist_
greater/user_greater) are now refused at open instead of read.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The clawhdf5-tools interop tests compare h5rs with h5ls, h5stat, h5dump and
h5diff, and CLAWHDF5_REQUIRE_INTEROP=1 turns a missing tool into a failure.
Also hold clawhdf5-tools to the no-C-in-the-default-build check.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
scripts/h5rs-fuzz.sh runs every h5rs subcommand over every file of a corpus
(default: the HDF Group's CVE reproducers), optionally with byte-flipped
copies (MUTATE=N), under a timeout and a memory limit, with a debug build so
integer overflow panics instead of wrapping; any exit status above 2 (a
caught panic, a timeout, a signal) fails it. It found size*8 overflows in
the datatype names on cve-2021-46244.h5, cve-2024-29161.h5 and unknown-1.h5
(fixed in the crate before it landed).
scripts/h5rs-check-ok-files.sh runs check (--data) over the conformance
files that clawhdf5 and h5py both read in full; none may be flagged.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
New workspace crate clawhdf5-tools with one binary, h5rs, built only on the
clawhdf5 facade and clawhdf5-format (no libhdf5, no C):
- ls [-r] [-v] FILE[/path]: h5ls's listing (same text in its first two
columns) plus the datatype; -v adds address, link count, layout and chunk
index, chunk size, storage, filters, datatype and attributes.
- dump [--json] [-A] [-p] [-d PATH] FILE: h5dump DDL (byte-identical to
h5dump 1.14.6 on the test files) or hdf5-json.
- stat FILE: h5stat's object/link/rank/layout/filter/attribute counts, raw
data and total size.
- diff [-r] [-q] [-d D] [-p R] A B [OBJ1 [OBJ2]]: structural and value
differences, exit 0/1/2 like h5diff.
- check [--data] FILE: walks every object, parses every message, verifies
the checksums of every v2+ structure (including the fractal heap blocks
the library never checks), checks chunk indexes against their datasets
and raw data for out-of-file or overlapping extents; every problem with
its address.
Values over --max-bytes are reported, not read; dense-storage heaps are
verified before objects are read from them; panics are caught (exit 3).
Tests compare with h5ls, h5stat, h5dump and h5diff and with h5py's values,
and flip the checksum of every checksummed structure in a v1.14-format file.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
A chunk dimension of 0 read a dataset as all fill values, 0x80000000 made
an 8 GiB chunk, and a chunk dimension the chunk index's offsets are not
multiples of read chunks at the wrong place (cve-2018-11205). libhdf5
refuses all of these; now so does clawhdf5:
- DataLayout::parse (H5O__layout_decode): no chunk dimension 0 ("bad chunk
dimension value"), at most 33 dimensions, and before layout v4 at least
2 ("bad dimensions for chunked storage"). New
FormatError::InvalidChunkDimensions.
- Reading a chunked dataset (H5D__chunk_init / H5D__chunk_set_sizes): the
chunk rank must match the dataspace's and a chunk must be under 4 GiB.
One chunked_read::chunk_geometry replaces the four copies of the rank
check.
- v1 B-tree chunk index (H5D__btree_decode_key): every key's offsets must
be multiples of the chunk dimensions, including the keys that only bound
a node, which is where cve-2018-11205's bad dimension shows. New
chunked_read::collect_chunk_info_checked; the chunked read and selection
paths use it.
New interop test header_validation_interop.rs: h5py writes chunked files
(layout v3 and v4), the script corrupts the chunk dimension, and
clawhdf5 must read exactly the copies h5py reads.
Conformance (cached corpus, tank): 570 ok, unchanged; cve-2018-11205 now
refuses the dataset h5py refuses; six more objects that already failed now
fail with libhdf5's reason.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
CHANGELOG (Unreleased): LZF, bitshuffle, bzip2 and Blosc read and write
in pure Rust, their features, the ChunkOptions::plugin field (breaking for
struct-literal construction), the filter registry, the named
UnsupportedFilter message, and Blosc2/ZFP still unimplemented.
README: the clawhdf5-format feature table gains lzf (default),
bitshuffle, bzip2, blosc and plugin-filters, with how to write them and
what is not implemented; no speed claims. docs/known-issues.md: the audit's
filter gap is marked fixed 2026-09-26 for LZF/bitshuffle/bzip2/Blosc,
Blosc2 and ZFP still open. CLAUDE.md: the clawhdf5-filters row no longer
says "No Blosc". clawhdf5-format's crate docs list the new features.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The check matches any *-sys crate, and clawhdf5-wasm pulls in js-sys,
wasm-bindgen's bindings to JavaScript, which compiles no C. Exempt it
by name so the check keeps catching real C for the wasm crate.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
scripts/ci-test.sh: the format feature matrix (clippy and tests) adds
plugin-filters; bitshuffle, bzip2 and blosc are each linted alone (blosc
and bitshuffle share code); the facade is linted with plugin-filters; and
the interop section runs tests/plugin_filters_interop.rs with it, against
h5py + hdf5plugin (CI's venv already installs hdf5plugin).
conformance/probe enables plugin-filters. Sweep (tank, 2026-09-26,
conformance/run.sh --no-fetch against the cached corpus): 573 of 697 ok
(baseline 569), no regressions; newly ok: h5ex_d_blosc.h5,
h5ex_d_bshuf.h5, h5ex_d_bzip2.h5, h5ex_d_lzf.h5. h5ex_d_blosc2.h5 and
h5ex_d_zfp.h5 remain UnsupportedFilter.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
libhdf5 1.14.4+ treats an integer, float or bit field wider than a byte
whose precision and offset leave more than half its bits unused as
corruption when the type sits in a header without a checksum (version 1),
unless the file is opened with H5Pset_relax_file_integrity_checks
(H5T_is_numeric_with_unusual_unused_bits). clawhdf5 read such types,
e.g. a 3-bit integer in 4 bytes (cve-2024-29162) or a 32-bit float in
65525 bytes (cve-2024-32614, tmisc38a.h5).
New Datatype::check_unused_bits (recursive) and Datatype::parse_in_header,
which applies it for version-1 headers. Dataset datatypes (facade File,
LazyFile, MmapFile; clawhdf5-io VOL, MPI VOL, async reader; the
conformance probe) and compact attributes in version-1 headers use it.
Conformance (cached corpus, tank): 570 ok, unchanged; cve-2024-29162,
cve-2024-32614 and tmisc38a.h5 now refuse the object h5py refuses, and
tmisc38b.h5 / unknown-1.h5 now fail with libhdf5's reason.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Blosc2 (32026) is out of reach for now: hdf5plugin's Blosc2 filter stores
each HDF5 chunk as a Blosc2 super-chunk frame (msgpack header, a compressed
chunk-offset index, trailer metalayers) and, for 2-D and larger chunks, as
a B2ND array whose n-D blocks have to be reassembled - on top of the Blosc2
chunk format itself (extended header, filter pipeline, special-value
chunks). ZFP (32013) is out of scope. Both keep failing with
UnsupportedFilter, and the message now says what the ID is:
"unsupported filter: 32026 (Blosc2, not implemented by clawhdf5)", or,
for a filter this build left out, "... (Blosc; this build lacks the
`blosc` feature)". filter_registry::known_filter exposes the table.
tests/plugin_filters_interop.rs: hdf5plugin's Blosc2 and ZFP datasets
read as an error naming the filter, never as data.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
hdf5plugin's Blosc (hdf5-blosc) failed with UnsupportedFilter(32001). The
new `blosc` feature decodes the Blosc 1 frame c-blosc 1.x writes: the
16-byte header, raw ("memcpyed") frames, the block table, blocks split
into one stream per byte plane (and the "do not split" flag), streams
stored raw, the byte shuffle and bit shuffle (whole 8-element groups, the
rest copied) - and every codec hdf5plugin offers: BloscLZ (implemented
here from c-blosc 1.21's blosclz_decompress, including its rejection of
malformed and truncated streams), LZ4/LZ4HC (lz4_flex), Snappy (snap),
Zlib (flate2) and Zstandard (ruzstd). Every stream must decode to exactly
its size and the frame to at most the chunk size; a frame of another
format version (Blosc 2) is a clear error.
It also encodes (DatasetBuilder::with_blosc(codec, level, shuffle)):
LZ4, Snappy, Zlib or Zstandard, with c-blosc's split rule, raw streams
where compression does not pay, and a stored frame for level 0 or
incompressible data. It cannot write BloscLZ (asking for it is an
error). `plugin-filters` enables LZF, bitshuffle, bzip2 and Blosc.
Interop: hdf5plugin writes all six codecs x {no, byte, bit} shuffle at
levels 5/9/1, plus level 0, over the 12-case matrix, read byte for byte;
our four codecs x four shuffle/level settings read back through
hdf5plugin. Both fail with the decoder removed. `cargo tree` with
`plugin-filters` has no -sys crate other than libbz2-rs-sys (pure Rust),
no cc and no cmake.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Datatype::parse now makes the checks of libhdf5's H5O__dtype_decode_helper
and fails with InvalidDatatype (libhdf5's own error text) instead of
decoding a corrupt type:
- size 0 ("invalid datatype size"), for every class;
- integer bit offset/precision outside the type, or precision 0;
- float sign/exponent/mantissa outside the type, empty, or overlapping;
normalization 3; bit 6 without bit 0 from version 3;
- compound with no members, a member outside the compound, a duplicate
name, or a member overlapping an earlier one;
- enum whose size differs from its base type's, or an empty member name;
- array of more than 32 dimensions or with a zero-sized one (v1 compound
array members now say so rather than InvalidDatatypeVersion);
- opaque tag length that is not a multiple of 8.
Bit 6 of a version-1/2 float's class bits used to be read as VAX order,
byte-swapping values; libhdf5 ignores it before version 3, and so does
this now.
Only checks HDF5 2.0 (h5py 3.16) makes are added: newer libhdf5 also
checks bit fields, the variable-length kind and array sizes, but h5py
opens files that fail those, so they are left out. Each check was
confirmed against h5py by corrupting a file it wrote.
The conformance probe now decodes committed datatypes, as h5py's f[name]
does. Conformance (cached corpus, tank): 570 ok, unchanged. Objects
libhdf5 refuses that clawhdf5 used to read: cve-2016-4332-mtime (/cmpnd),
cve-2017-17508, cve-2024-32616 (/type1), cve-2024-32618, cve-2026-34734,
bad_compound.h5 (/cmpnd, /dataset); eight more that already failed now
fail with libhdf5's reason (e.g. cve-2024-29163 "mantissa range out of
bounds").
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Changelog entry with the sizes measured on tank on 2026-09-26 (and the
h5wasm 0.10.3 comparison), the browser build's limits as a known-issues
entry, and where its tests run.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
cargo test --workspace unifies clawhdf5-format/zstd on (another member
enables it), so the native interop test read the Zstd dataset that the
wasm build refuses. The fixture now records its values plus the error
the wasm build must give; the native test accepts either, the Node test
of the real wasm package still requires the error.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Drop a file (or pass ?file=<url>&path=<object>), browse the tree lazily,
see a dataset's type, shape, max shape and attributes, and page through
its values as 50x12 hyperslab windows (leading dims of 3-D+ data held
at chosen indices). build.sh produces pkg/ (not committed) with
wasm-bindgen --target web and checks the CLI matches the crate version.
test/run.sh builds it and runs test.mjs under Node against the h5py/
netCDF4 fixture (250 checks: every dataset whole and as a strided
hyperslab, listings, attributes, error paths, the page's DOM-free
helpers), then browser.sh renders the page in headless Chromium for
eight objects and checks the DOM. The fixture gains LZ4 (read) and Zstd
(refused: links C) datasets and a compound attribute (value null plus
its type). ci-test.sh runs it when node and wasm-bindgen exist; the CI
container has neither, so CI relies on the native h5py_interop test.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
hdf5plugin's BZip2 failed with UnsupportedFilter(307). The new `bzip2`
feature decodes the single bzip2 stream H5Zbzip2.c stores, bounded by the
chunk size (a truncated stream is an error, not short data), and encodes
at block size cd_values[0] (DatasetBuilder::with_bzip2(level)). It uses the
bzip2 crate's default backend, libbz2-rs-sys, a pure-Rust port of
libbzip2: `cargo tree` shows no cc/cmake, and nothing is compiled from C.
Interop: hdf5plugin writes block sizes 9, 1 and 5+shuffle over the
12-case matrix, read byte for byte; ours at 9 (shuffled) and 1 (not)
reads back through hdf5plugin. Both fail with the decoder removed.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
hdf5plugin's Bitshuffle failed with UnsupportedFilter(32008). The new
`bitshuffle` feature (pure Rust: lz4_flex, and ruzstd for Zstandard — the
`zstd` feature's libzstd is not needed) decodes all three modes of
bshuf_h5filter.c — transpose only, LZ4 and Zstandard blocks behind the
12-byte header — including the default and explicit block sizes, the
shorter last block rounded down to a multiple of 8 elements, and the
untransposed trailing elements. Sizes read from the chunk are bounded by
the chunk size.
It also encodes: DatasetBuilder::with_bitshuffle(BitshuffleCompression)
or PluginFilter::Bitshuffle { block_size, compression } writes the filter
with hdf5plugin's cd_values and no automatic byte shuffle. ruzstd has one
compression level (about zstd's 1); the requested level is recorded.
The bit transpose is checked bit for bit against a one-bit-at-a-time model
(which matched hdf5plugin's output) and is shared with blosc next.
Interop: hdf5plugin writes none/LZ4/Zstandard at default and explicit
block sizes and levels over the 12-case matrix, read byte for byte; our
three modes at two block sizes read back through hdf5plugin. Both fail
with the decoder removed.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
ObjectHeader::parse now checks each header message the way libhdf5's
H5O__chunk_deserialize does, and fails with InvalidObjectHeader (libhdf5's
own error text) instead of reading objects out of a corrupt header:
- v1: every message in chunk 0 is read (not just the prefix's count) and
more messages than the prefix claims is "bad object header message
count"; message sizes must be multiples of 8; leftover bytes are a gap,
which only v2 allows; the prefix's chunk size must fit its count.
- v1 and v2: a message running past its chunk is an error (it used to end
the chunk quietly, dropping it and everything after); contradictory
message flags; a message of a class that cannot be shared flagged
shared/shareable; a reference-count message in a v1 header; malformed
continuation, reference-count and modification-time messages (libhdf5
decodes these while loading the header).
- v2: unknown header status flags, max_compact < min_dense, a chunk 0
smaller than a message header, a gap in a chunk that has NIL messages.
Conformance (cached corpus, tank): 569 -> 570 ok (h5stat_err_refcount.h5).
Objects libhdf5 refuses that clawhdf5 used to read: cve-2016-4332-mtime
(/dataset), cve-2016-4332-mtime-new, cve-2018-11204, cve-2018-13873,
cve-2024-32619, cve-2024-33873, cve-2024-33874, gh-4433-poc-08; seven more
CVE objects that already failed now fail with libhdf5's reason.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
h5py's built-in compression="lzf" failed with UnsupportedFilter(32000). The
new `lzf` feature (no dependencies, on by default in clawhdf5-format and
the facade) decodes the raw liblzf stream h5py's filter stores, bounded by
the chunk size, and encodes it: DatasetBuilder::with_lzf() (or
with_plugin_filter(PluginFilter::Lzf)) writes the filter with h5py's
cd_values (filter version 4, liblzf 0x0105, chunk size in bytes), flagged
optional as h5py does. ChunkOptions gains a `plugin` field for the plugin
filters; build_pipeline_for_chunk passes the chunk size to filters that
record it.
tests/plugin_filters_interop.rs: h5py writes LZF (alone, with shuffle, with
shuffle+fletcher32) over 12 dtype/shape/chunk/data cases with partial edge
chunks and incompressible data, and every dataset reads byte for byte equal
to its unfiltered twin; our LZF output (1-D and 2-D, edge chunks, with and
without shuffle) reads back in h5py. Both fail with the decoder removed.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
open(bytes) -> H5File with kind/list/info/attrs/attrErrors/read/
readHyperslab. Numeric data comes back in the typed array of the
stored width (Int16Array for i16, BigInt64Array for i64, Float32Array
for f32/f16, ...), strings and enum names as string arrays, array
datatypes flattened with their dims appended to the shape. Compound,
reference, opaque and VL-sequence datasets are refused with an error
naming the type; nothing is returned as reinterpreted bytes.
The logic is in a plain-Rust core module, tested natively: unit tests,
and h5py_interop, which compares every dataset, hyperslab, listing and
attribute of an h5py- and a netCDF4-written file with what libhdf5
reads back (generator shared with the Node test of the built package).
No mmap, no threads; lz4 is on, zstd/szip (C) are not. A
wasm-release profile (opt-level s, LTO) serves the browser build.
ci-test.sh lints the crate for wasm32 and checks it builds no C.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The facade already builds for the browser target without mmap (and
with it: memmap2 compiles there and File::open just fails, as std::fs
does). Nothing needed gating; keep it that way with a ci-test.sh step,
and install the target in the CI container.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
decompress_chunk_masked and compress_chunk matched on the filter ID. They
now look the ID up in filter_registry: a static table of the built-in
filters compiled into this build (a filter whose cargo feature is off is
simply absent), then the codecs an application registered at run time with
register_filter (a FilterCodec, or a plain decoding closure). Registered
codecs cannot shadow a built-in one, and their output is held to the same
per-stage bound as the built-in decoders. An ID in neither tier still fails
with UnsupportedFilter(id).
The "feature off" stub functions that returned UnsupportedFilter are gone:
the table leaves those filters out instead.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
concurrent_read reads one shared File from 1-16 threads: every dataset
in full (distinct datasets per thread) and random hyperslabs of one
dataset, over a deflate and a contiguous file it generates (or reuses
while manifest.json matches). It reports decoded MB/s and scaling
efficiency, warm or --cold (posix_fadvise) page cache, sizes the decode
pool with --decode-threads, and writes JSON.
scripts/concurrent_read_h5py.py runs the same workload on the same files
with h5py threads or spawned processes (same splitmix64 data and slab
stream, checked at spot elements), and compare_concurrent_read.py prints
one table and refuses runs with different workloads. A smoke test runs
all three end to end on tiny files (h5py half honours CLAWHDF5_PYTHON /
CLAWHDF5_REQUIRE_INTEROP).
BENCHMARKS.md gets a "Concurrent reads" section with the commands, marked
not yet measured.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
dtype() simplifies the type (no byte order, string padding or member
offsets), so callers could not decode read_selection's bytes for types
the typed read_* methods skip. raw_datatype() returns the parsed
Datatype, committed types resolved, for use with data_read.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Regenerated on tank at 10d1029: ok 467 -> 569, our-error 123 -> 14,
mismatch 15 -> 22 (six user-defined-link files moved from our-error to a
listing difference), no panics, hangs, crashes or OOM. Baseline raised.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Superblock::parse now refuses a user-block offset, and the raw read path
no longer guesses a VDS fill value. The probe looks at the file from the
superblock on and reads virtual datasets with vds::read_virtual_dataset,
the dataset's fill value and its source-derived extent.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The test compared h5py against its own expected table, so it passed with
the fix reverted. Found by the adversarial review.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Group::attrs now leaves out an attribute it cannot decode. The agent read
its settings through it, so a store whose float16 (or compression,
quantized_index, WAL mark, signature...) attribute could not be decoded
opened with the default in its place, and no error. /meta is now read
with attrs_with_errors and any unreadable attribute is a Schema error,
as it was before attrs became tolerant. Found by the adversarial review.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Their elements are global-heap IDs and object addresses in the source
file. The VDS reader copied them raw, so anything decoding them against
the virtual dataset's file got another object's data with no error.
Same-file sources are unaffected. Found by the adversarial review.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Unlimited VDS mappings were refused, and printf-style source names
("f-%b.h5") were not expanded, so those regions read as fill (read-matrix
case 0470: 29 of 30 values wrong). All 7 virtual datasets in the libhdf5
test set use such mappings.
Implement H5Dvirtual.c's semantics in the vds module:
- %b is the block number, %% a literal %, other specifiers are an error;
block j of the virtual selection comes from the source named with j,
probing from 0 to the first missing source (printf gap 0);
- unlimited source/virtual selections are clipped to what the source's
current extent fills (H5S_hyper_get_clip_extent_match, partial last
block included);
- the extent is recomputed as H5Dget_space does (view "last available":
the largest clip, never below what limited mappings need), exposed as
vds::virtual_dataset_extent and used by Dataset::shape();
- a source in the other byte order is byte-swapped; other conversions stay
an error.
Tests: vds_interop::vds_printf_source_names,
vds_unlimited_mappings_follow_source_extents (h5py low-level API, earliest
and latest format) and vds_libhdf5_test_files (vds-eiger, 4_vds and
vds-percival-unlim-maxmin from HDF5's tools/test/testfiles/vds, committed
as fixtures) all compare shape and values with h5py; unit tests for the
clip arithmetic, name parsing and mapping rules.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
A message shared through the file's SOHM heap (H5Pset_shared_mesg_index)
is referenced by heap ID, which needs the SOHM table from the superblock
extension. Only message_data_with_sohm (used for fill values) loaded it;
resolve_shared_message passed no table, so a SOHM-shared datatype,
dataspace, filter pipeline or attribute failed with "invalid shared
message version: 2" and the dataset or attribute could not be read.
resolve_shared_message now loads the table when the reference carries a
heap ID.
Found while making attrs() tolerant: SOHM attributes turned from an
error into missing keys in the audit read matrix. With this fix all 36
SOHM cases there match h5py (datasets, fill values and attributes, every
shareable message type, libver earliest and latest).
Regression test: sohm_shared_messages_resolve (h5py writes files sharing
each message type on its own and all of them; values and attributes
checked).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
libhdf5 walks a local heap's free list when it loads the heap's data and
refuses the heap ("bad heap free list") when a free block starts or ends
outside the data segment, or links to offset 0. We never looked at the
free list, so a damaged old-style group listed names read from the broken
heap: once the user block of cve-2021-36977.h5 was applied, its root
listed eight garbage names where libhdf5 fails.
LocalHeap::validate_free_list (new) mirrors H5HL__fl_deserialize, with a
cycle bound, and accepts H5HL_FREE_NULL (1) or an all-ones head as the
end of the list. Like libhdf5 it runs when the first name is needed, not
on parse, so an empty group with a damaged heap still lists as empty
(cve-2018-13871.h5, cve-2024-29166.h5, gh-4431-poc-03.h5 keep matching
h5py).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
A file may start with a user block (h5py userblock_size, h5jam), putting
the superblock at 512, 1024, ...; every address in the file is then
relative to the superblock. The signature search found it, but every
reader passed the whole file to the parsers, so addresses landed
userblock bytes early and the root group failed with
InvalidObjectHeaderVersion (twithub.h5, twithub513.h5,
h5clear_fsm_persist_user_*.h5).
Readers now view the file from the superblock on, taking the signature's
position as the base address as libhdf5 does: File (mmap, buffered,
from_bytes), MmapFile, LazyFile, AsyncHDF5File, the VOL and MPI VOL
readers, the HNSW loader and external VDS source files. File, MmapFile
and LazyFile gain user_block_size(). The new signature::split_user_block
returns the two parts, and Superblock::parse refuses a non-zero offset
(UserBlockNotStripped) so a format-level caller cannot silently apply
superblock-relative addresses to the whole file.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
A version-1 shared message (HDF5 1.6) embeds the target as a symbol-table
entry: after six reserved bytes comes a length-sized local-heap offset,
then the object header address. We read the heap offset as the address,
so datasets using a committed datatype in 1.6-era files (tcompound.h5,
tcompound2.h5) failed with InvalidObjectHeaderVersion. parse_shared_ref
now takes length_size and skips the offset, as libhdf5 does.
Resolving a reference also no longer falls back to the first message of
any type in the target header: a missing target message is
SharedMessageTargetMissing instead of garbage.
Fixture: tcompound.h5 from libhdf5's tools/test/testfiles (8 KiB).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
HDF5 1.6 encoded a compound member that is a fixed-size array through
legacy per-member fields (dimensionality, permutation, four dimension
sizes) that the v1 decoder skipped, so a [4] i32 member read as one i32
with the wrong size. Build the array type from those fields as libhdf5
does (ignoring the permutation) and refuse more than four dimensions.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Runs conformance/run.sh in rust:latest on a schedule and on demand, with its
own venv (pinned h5py/numpy/hdf5plugin/netCDF4) and hdf5-tools. Fails on any
panic, hang, crash or OOM in clawhdf5 and on a drop against
conformance/baseline.json; prints CONFORMANCE.md into the job log and
uploads nothing. Plain git checkout, no JavaScript actions.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
conformance/run.sh fetches eight public HDF5 corpora pinned by commit
(conformance/corpus.txt) into a gitignored cache, reads every file with
clawhdf5 (conformance/probe, a crate outside the workspace) and with
h5py/libhdf5 (ref.py), and the CVE files with h5dump, each under a timeout
and an address-space limit; compare.py classifies the files, report.py
writes CONFORMANCE.md and check.py gates on panics/hangs/crashes/OOM and on
regressions against conformance/baseline.json. ~25 s once cached.
Changes from the ad-hoc audit harness:
- the probe compares non-IEEE-layout floats (N-Bit) and integers with a bit
offset or reduced precision as the values libhdf5 converts them to, not
raw file bytes: 8 files that showed as mismatches now read identically;
- ref.py exits without tearing down h5py objects: libhdf5 2.0 aborts while
freeing them for two files about half the time, which flipped them
between ok and h5py-cannot-read from run to run;
- the file list is defined (list_files.py): netCDF classic files are left
out, 11 HDF5 files the ad-hoc sweep missed are in.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Elements of a virtual dataset that no mapping supplies (unmapped regions,
a missing source file, a missing source dataset) read as 0 instead of the
fill value libhdf5 returns — silent wrong data for any VDS created with a
non-zero fillvalue (read-matrix cases 0471/0472: -1 and 7 read as 0). A
missing source dataset was an error; libhdf5 reads it as fill.
Move VDS assembly into a new vds module following H5Dvirtual.c:
vds::read_virtual_dataset takes the dataset's fill value and a
VdsFileResolver that can refuse a name, and reports how many elements were
unmapped. Sources are read with their own fill value, and a source whose
datatype differs from the virtual dataset's is an error (libhdf5 converts).
File passes the dataset's fill value, resolves source names against the
virtual file's directory, and refuses names that leave it with an error
instead of reading them as fill. read_selection on a VDS goes through the
same fill-aware path.
The raw-read API (read_raw_data_full*) has no fill value, so it now errors
for a VDS with unmapped elements instead of guessing zeros.
Tests: vds_interop::vds_unmapped_regions_read_as_fill_value (external,
same-file, missing file/dataset, sparse source with its own fill, int
fill; earliest and latest format) and
vds_source_outside_directory_is_an_error_not_fill, both against h5py;
integration_test::v4_virtual_dataset_raw_api_refuses_to_guess_the_fill_value.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
attrs() read every attribute of an object through extract_attributes_full,
so one attribute it could not read (a corrupt or unsupported attribute
message, or a heap object it could not locate) failed all of them — the
same shape as the huge-object bug, where one 8 KiB attribute hid every
attribute on a NetCDF file's root group.
- clawhdf5-format: new attribute::extract_attributes_tolerant returns the
attributes it could read plus one error per attribute it could not.
Errors in the attribute index itself (Attribute Info message, dense
heap header, B-tree) still fail, since then it is unknown which
attributes exist. extract_attributes_full is unchanged (strict); both
share one implementation.
- clawhdf5: attrs() on Group/Dataset, MmapGroup/MmapDataset and
LazyGroup/LazyDataset leaves an unreadable attribute out (documented),
and the new attrs_with_errors() returns the map with the per-attribute
errors. A value is either returned complete or not at all.
Regression test: one_unreadable_attribute_does_not_hide_the_others (h5py
writes 11 dense attributes; one message's version byte is corrupted;
before: attrs() failed with InvalidAttributeVersion(127), after: the 10
others come back with their values and one error is reported).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Group::datasets()/groups() (and the Mmap/Lazy handles) listed only hard
links, so a soft link to a dataset or group was missing, and dataset(name)
/ group(name) on a group handle could not open one. In old-style (symbol
table) groups a soft link's entry has no object header address, and the
listing failed outright trying to parse one.
The three facade handles each had their own copy of the child-listing
code; they now share group_v2::resolve_group_children, which returns hard
links plus soft links resolved to their targets (relative targets from
the group holding the link, via the new resolve_path_from). A dangling or
cyclic soft link, an external link and a user-defined link are left out —
h5py lists their names but cannot open them. Any other error met while
resolving is returned, not hidden.
Path resolution now walks a relative soft link's target from the group
holding it instead of rebuilding the path from the root (same result,
one less re-walk), and ignores "." components.
Regression test: soft_links_are_listed_as_their_targets (h5py writes
absolute, relative, group, dangling, cyclic and external links with
libver latest and earliest; listings compared with h5py for File,
MmapFile and LazyFile).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
With a 2.0 low version bound, libhdf5 stores the VDS mapping list as heap
block version 1: every entry starts with a flags byte (0x04 same file, no
file name; 0x01/0x02 file/dataset name shared with an earlier entry, whose
index is stored in place of the name). The parser treated only a leading
0x04 byte as special, so a 0x00 flags byte read as an empty (same-file)
name and shared names were read as garbage.
Decode it as H5D__virtual_load_layout does, refusing unknown flags,
forward references and block versions above 1.
Test: vds_interop::vds_mapping_block_version1_shared_names (h5py
libver=("v200","v200") with repeated long names; failed before with
"unknown dataspace selection type") plus the exact heap block as a unit
test.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
libhdf5 serializes a VDS hyperslab as version 1 (irregular, 4-byte block
corners) for the default format bounds, and as version 2 (regular, 8-byte)
for unlimited selections in the 1.10 format. Only version 3 was accepted,
so every h5py VDS written with default libver failed with "only version-3
hyperslab selections are supported" (5 libhdf5 test files in the sweep).
Decode all three versions following H5S__hyper_deserialize, including
irregular hyperslabs (a union of blocks, enumerated in row-major order as
libhdf5 iterates them) and the all-ones "unlimited" count/block marker.
SerializedSelection exposes the raw form for unlimited-mapping support.
Test: vds_interop::vds_version1_irregular_hyperslab_selections compares
default-libver h5py VDS reads (contiguous, strided and 2-D block mappings)
with libhdf5's values.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Link types 65-255 are user-defined: their target is only meaningful to
the application that registered the link class. LinkMessage::parse
rejects them with InvalidLinkType, and group traversal propagated that,
so one such link made the whole group unlistable and every path through
it unresolvable (libhdf5's tall.h5 and tudlink.h5, class 187).
Group traversal (compact and dense) now leaves user-defined links out,
the way h5py leaves out links it cannot open; reserved types (2-63) are
still an error.
Regression test: user_defined_links_do_not_break_the_listing, on
libhdf5's own tools/test/testfiles tall.h5 and tudlink.h5 (BSD-style
HDF5 licence, 10 KB and 1 KB), committed as fixtures.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
A heap ID's type is in bits 4-5 of its first byte (H5HF_ID_TYPE_MASK
0x30); bits 6-7 are the ID version. The reader took the type from bits
6-7, so every huge object ID (0x10) was decoded as a managed one and
failed — and since dense attributes are read all at once, one attribute
over the heap's 4 KiB managed limit made every attribute on its object
unreadable (netcdf4-python's issue671.nc / issue672.nc).
- Huge objects (type 1): located directly from the ID when address and
length fit in it, otherwise through the huge-object v2 B-tree (record
types 1 and 2); filtered huge objects are decoded with the heap's
pipeline and their filter mask.
- Tiny objects (type 2): read from the ID itself.
- Filtered heaps: the header's pipeline is parsed (it was skipped short,
so the header checksum was read from the wrong place), indirect-block
entries for direct blocks carry their filtered size and mask, and
direct blocks are decoded before objects are read from them.
- An unknown ID version is an error.
FractalHeapHeader gains huge_btree_address, filter_pipeline,
root_direct_block_filtered_size, root_direct_block_filter_mask,
offset_size and length_size; read_managed_object now accepts any ID type.
Regression tests (h5py-written, compared with h5py):
dense_attribute_stored_as_a_huge_heap_object, dense_group_with_a_huge_link,
dense_group_with_a_filtered_link_heap; unit tests
tiny_object_is_read_from_the_id, huge_object_with_a_direct_id,
unknown_heap_id_version_is_refused.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
A version-1 shared message reference is version, type, six reserved bytes
and then an old-style symbol table entry: link-name offset (length size),
object header address, cache type, reserved, scratch. We read the address
straight after the reserved bytes, i.e. the link-name offset, and the
committed datatype lookup failed with InvalidObjectHeaderVersion (the bytes
checked in tcompound.h5: name offset 0x10, then 0x590 = /type1). Datasets
of 1.4/1.6-era files that use a committed datatype were unreadable.
Skip the name offset. parse_shared_ref has no length size, so add
parse_shared_ref_sized and use it in every internal caller;
parse_shared_ref keeps its signature and assumes length size == offset
size. The old parse_v1_ref unit test encoded the wrong layout and now uses
the real bytes.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Compound datatype version 1 carries, per member, a dimensionality and four
dimension sizes (HDF5 before 1.4 had no array class). The parser skipped
those 28 bytes, so a member such as `f: f32[4]` came back as a single f32
at the member's offset: the compound's size was right but its members were
wrong. libhdf5 wraps such a member in an array type of the first
`dimensionality` sizes and ignores the permutation; do the same, and
reject a dimensionality above 4 as libhdf5 does.
Only files old enough to also use layout message v1 have these, so this
became reachable with the previous commit (tarrold.h5, tcompound.h5).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
HDF5 1.4/1.6-era files store the layout as version 1 or 2: version,
dimensionality, class, 5 reserved bytes, an address (contiguous and chunked
only), dimensionality 32-bit sizes (with the trailing element-size
dimension) and, for compact storage, a 32-bit size and the raw data. They
failed with InvalidLayoutVersion — 84 of the 686 files in the audit sweep,
205 datasets.
Map them onto the existing variants: chunked uses the same version-1
B-tree chunk index as version 3 and is reported as version 3, so every
chunked read path (filters, selections, caches) applies unchanged.
Contiguous size is the product of the stored dimensions, which is what
libhdf5 computes from the dataspace; a disagreement fails the reader's size
check instead of returning wrong data.
Fixtures are HDF5's own deflate.h5 (v1, chunked + deflate) and
h5ex_g_iterate.h5 (v2, contiguous, one unallocated dataset); the new
interop test compares every dataset byte for byte against h5py.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
An internal node's child pointer is an address, the child's record count
and (below the first internal level) the child subtree's total record
count. libhdf5 (H5B2__hdr_init) encodes the record count in the width of
a leaf's maximum and the subtree total in the width of cum_max_nrec for
that depth, computed level by level from the node size. The reader
guessed 2 * leaf_max and leaf_max^depth, which agree at depth 2 but not
at depth 3: a 24 000-link group's name index has depth 3, its root's
pointers were read 3 bytes wide instead of 2, and listing failed with a
garbage heap offset.
Regression tests: dense_group_with_a_three_level_name_index (h5py writes
24 000 links; listing compared with h5py) and
subtree_capacity_matches_libhdf5.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
A child indirect block in row r of a fractal heap's doubling table spans
that row's block size of heap space, so it has
log2(size) - log2(start_block_size * width) + 1 rows (libhdf5's
H5HF__dtable_size_to_rows). The reader used row - first_indirect_row + 1,
which undercounts, so every object stored past the root block's direct
rows (512 KiB with libhdf5's defaults) was unreachable: dense groups with
a few thousand long link names, or ~20 000 short ones, could not be listed.
Regression test: dense_group_whose_heap_outgrows_the_root_direct_rows
(h5py writes 2 500 links with 248-byte names; listing compared with h5py).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
CHANGELOG: upgrade notes (changed read results for max-shape files,
saturating conversions, new writer errors, format-crate API changes) and
the reader/writer correctness fixes. known-issues: the silent-wrong-data
table with before/after sweep numbers, the gaps still open, and a
correction to the Extensible Array entry, which said files we wrote were
unaffected. CLAUDE.md: clawhdf5-gpu is vector distance computation, not
I/O, and clawhdf5-filters holds only deflate backends (no Blosc).
Also a facade test that libhdf5's 20-bit N-Bit float test data reads as
libhdf5's values.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The interop step built writer_h5py_tests without the lz4/zstd features,
so the hdf5plugin round-trips added with the registered LZ4 framing and
the Zstd content-size fix never compiled in CI, and CI never installed
hdf5plugin.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The maxshape checks added to chunked_write use format!, which a no_std
build has to import from alloc (scripts/check-nostd.sh).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
dataset_fill_value treated a shared Fill Value message as "no fill
value", so unwritten storage of a dataset whose fill value lives in the
file's shared-message (SOHM) heap read as zeros rather than its fill
value. libhdf5 shares fill values whenever the file has a SOHM index for
them.
- fill_value::dataset_fill_value_in follows the reference (another object
header, or the SOHM heap); read_full_with_fill and the facade's
selection read use it.
- dataset_fill_value, which has no file to follow a reference into, now
returns UnresolvedSharedMessage for a shared message instead of None.
- shared_message::load_sohm_table / message_data_with_sohm load the SOHM
table from the superblock extension on demand.
- parse_sohm_table skipped each index's leading version byte, reading
every field one byte off; SOHM references could never resolve.
Fixture shared_fill_value.h5 (HDF5 2.0, gen_shared_fill.py): sohm_b read
[0,1,2,3,0,0,0,0] and now reads [0,1,2,3,-7,-7,-7,-7], as h5py does.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
libhdf5 applies filters in pipeline order, so with Fletcher32 before
deflate (h5repack_filters.h5 /dset_all: shuffle, fletcher32, deflate; or
h5py's set_fletcher32() then set_deflate()) the compressor holds the
chunk plus a 4-byte checksum. decompress_chunk bounded every stage by the
chunk size and rejected it: "deflate: output exceeds size limit". Bound
each stage by the chunk size plus 4 bytes per Fletcher32 that precedes
it in the pipeline.
Test: fletcher32_before_deflate_decodes (h5py-written chunk, and our own
shuffle + fletcher32 + deflate round trip); failed before.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Any maxshape forced chunked storage, even one equal to the shape, which
cannot grow. h5py and the library store such a dataset contiguously; we
now do too unless chunks (or a filter) are requested.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
D-scale floats were rebuilt as `minval + code / 10^D` in f64 and then
rounded to f32 once, but libhdf5 (H5Z_scaleoffset_modify_3/4 with
`float`/`powf`) computes `(float)(int)code / powf(10, D) + min` in single
precision. The two differ by 1 ULP for some values: le_data.h5
/Scale_offset_float_data_{le,be} gave 1.6663332 (0x3fd54a69) where
libhdf5 gives 1.6663333 (0x3fd54a6a). Use f32 arithmetic for 4-byte
floats and `(double)(long)code / pow(10, D) + min` for 8-byte ones.
Test: scaleoffset_float_dscale_matches_libhdf5_bits (le_data.h5 float
LE/BE and double chunks, bit-exact against h5py); failed before.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
A dataset with more than one unlimited dimension got an Extensible Array
index, which libhdf5 refuses ("already found unlimited dimension"), so
the whole file failed to open in h5py and h5dump. The previous commit
turned that into a write error; this one writes what the library itself
uses there: a version-2 B-tree chunk index (record type 10/11), as a
single leaf of the library's 2048-byte node size, or a larger leaf when
the records do not fit. The root's record count is 16-bit, so more than
65535 chunks is still refused rather than written wrong.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The object header parser failed on an unknown message with flag bit 3
set and ignored bit 7. Per the spec, bit 3 means "fail if unknown and
the file is opened for writing" and bit 7 "fail if unknown, always".
The parser only reads, so it now ignores bit 3 (as libhdf5 does for a
read-only open) and refuses bit 7, in v1 headers, v2 headers and their
continuation chunks.
On libhdf5's conformance file tbogus.h5 (added as a fixture) we used to
refuse Dataset2 and open Dataset3; we now match libhdf5: Dataset1, 2, 4
and 5 open, Dataset3 is refused.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
- libhdf5 sets cd_values[1] ("need not compress") when every field is
already full width and then stores the chunk unchanged
(H5Z__filter_nbit: `if (cd_values[1]) HGOTO_DONE`). We ignored it and
tried to unpack, so tfilters.h5 / h5stat_filters.h5 `/all` (shuffle +
szip + deflate + fletcher32 + N-Bit) failed with "nbit: packed data too
short". A type with no N-Bit parameters (cd = [3, 1, nelmts]) is now
accepted the same way.
- Class 4 (H5Z_NBIT_NOOPTYPE: enum, string, opaque, ... members) is
stored whole, 8 bits per byte; it was UnsupportedFilter(5)
(h5repack_nested_8bit_enum_deflated.h5).
N-Bit on floats was not wrong in the filter: for le_data.h5 /
Nbit_float_data_* our output equals libhdf5's decoded bytes in the file
datatype (a 20-bit float, offset 7, bias 31). h5py's values differ
because libhdf5 then converts that custom float layout to IEEE, which
our datatype reader does not do; nbit_float_matches_libhdf5_file_type_bytes
pins the filter output and the doc comment says where conversion belongs.
Tests: nbit_need_not_compress_is_passthrough,
nbit_in_multi_filter_pipeline_matches_libhdf5 (tfilters.h5 chunk, szip
feature), nbit_compound_with_enum_member_matches_libhdf5 all failed
before; nbit_float_matches_libhdf5_file_type_bytes (guard).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
FileWriter writes the root group plus one level of groups; it has no way
to create intermediate groups. create_group("a/b") therefore stored a
single link literally named "a/b", which no HDF5 reader can resolve
(h5py: "component not found"). Nesting would mean restructuring the
writer's layout around a group tree, so for now finish() rejects any
group, dataset or external-link name that is empty, "." or contains '/'.
Attribute names may still contain '/'.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
A File is Send + Sync and keeps one ChunkCache for all its datasets.
The cached readers bound that cache to "the current dataset" with
ensure_dataset(addr), then checked, built and read its index and its
decompressed chunks in separate lock acquisitions. Two threads reading
two chunked datasets interleaved those steps, so one could store its
chunk index under the other's binding, or get the other's decompressed
chunk for the same coordinate: wrong data, or an index-out-of-bounds
panic when the ranks differed (16 threads x 40 reads over 24 datasets
panicked on every run).
The cache now keeps per-dataset state keyed by chunk-index address:
the chunk index, ChunkIndex and ChunkLayout per dataset (held as Arcs,
built outside the lock, first writer wins), and decompressed chunks
keyed by (address, coordinate). The chunked readers use the new
addr-taking methods (chunks_for, chunk_layout_for, get/put_decompressed_in,
prefetch_hint_in) exclusively. Memory stays bounded: decompressed data by
the existing byte/slot budget across datasets, indexes by at most 64
datasets and 2^20 index entries in total, dropping the least recently
used dataset's index first. Switching datasets no longer throws away the
other datasets' cached chunks.
The address-less methods remain and act on the dataset last bound with
ensure_dataset; they are documented as not for concurrent readers.
Regression: threads_reading_different_datasets_get_their_own_chunks
(crates/clawhdf5/tests/concurrent_chunk_cache.rs), plus cache unit tests
datasets_sharing_coordinates_stay_separate, dataset_indexes_are_bounded
and concurrent_readers_of_different_datasets_see_their_own_chunks.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The writer indexed chunks by their position in the current shape, the
same mistake the reader had. With a finite maxshape larger than the shape
the Fixed Array was sized for the shape, so libhdf5 looked up chunks past
its end ("addr overflow"); with the unlimited dimension anywhere but first,
e.g. maxshape (20, None), libhdf5 swizzles that dimension to the slowest
position and read our Extensible Array scrambled. Two unlimited dimensions
produced a file libhdf5 refused to open ("already found unlimited
dimension").
Chunks are now placed with the shared chunk_grid linearisation: Fixed
Array slots cover every chunk of the maximum extent (unwritten ones
undefined), Extensible Array indexes are swizzled, Single Chunk is only
used when the maximum extent is one chunk, and a maxshape that is smaller
than the shape, has more than one unlimited dimension, or would need an
absurd Fixed Array is an error instead of a bad file.
build_chunked_data_from_precompressed now returns a Result.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
An empty AttrValue::String (or a StringArray of empty strings) was
written with a size-0 fixed-length string type. libhdf5 rejects that
("invalid datatype size"), and the failure takes every attribute on the
object with it. Strings are now at least 1 byte, NUL-padded, which is
how h5py stores "" and reads back as "" in both h5py and our reader.
check_encodable also refuses a size-0 string type passed in directly.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
SZIP-filtered datasets from libhdf5 came back as garbage or zeros with no
error (ref_szip.h5, h5repack_szip.h5, noencoder.h5, le_data/be_data
Szip_float_data_*), and 64-bit ones failed with "invalid bits per
sample" (h5wasm compressed.h5). The decoder called aec_buffer_decode
directly, but libhdf5 goes through szlib's SZ_BufftoBuffDecompress
(H5Zszip.c), which libaec implements with reshaping (sz_compat.c).
Differences, all fixed:
- H5Zszip.c prefixes the stream with the 4-byte LE uncompressed size; it
was fed to libaec as data.
- 32- and 64-bit samples are coded as byte planes of 8-bit samples and
must be de-interleaved.
- The reference sample interval is ceil(pixels_per_scanline /
pixels_per_block), not a fixed 128.
- Scanlines that are not a whole number of blocks are padded and must be
unpadded.
- Byte order comes from the MSB option bit; LE data was decoded as MSB.
Test: szip_decodes_libhdf5_chunks_exactly compares chunks from HDF Group
test files (noencoder.h5, le_data.h5) and an h5py-written file (64-bit,
16-bit, padded scanlines, NN and EC) byte for byte with h5py's values;
it failed before on the first case.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
FillTime::to_byte had the fill-time field rotated against libhdf5
(H5D_FILL_TIME_ALLOC = 0, NEVER = 1, IFSET = 2): Never was written as
ALLOC, Alloc as IFSET and IfSet as NEVER, as h5py reported. The flags
byte is now late allocation plus the right code, and FillTime::from_byte
decodes it.
The default becomes IfSet, which is libhdf5's default and exactly the
byte (0x0a) every dataset was already written with, so default output
does not change; `Alloc` was documented as the C library's default but
never was. DatasetCreateProps follows.
DatasetBuilder::with_fill_value sets a user-defined fill value (one
element's stored bytes, checked against the datatype size), written as a
defined value in the fill value message. h5py reports it, and extending
the dataset in h5py fills the new elements with it.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
FileWriter::with_page_size wrote a "version 4" superblock with an extra
page-size field. HDF5 has no superblock version 4, so libhdf5 refused
every such file ("bad superblock version number").
A paged file is now what libhdf5 itself writes for fs_strategy="page":
a v3 superblock whose extension object header holds a File Space Info
message (strategy PAGE, the page size, free space not persisted; same
bytes and flags as HDF5 2.0), with the file padded to a whole page.
h5py opens it, reports the strategy and page size, and can modify it in
r+ mode. Page sizes outside libhdf5's 512 B..1 GiB are an error.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The Extensible Array writer only filled the index block's 4 inline
elements and the 6 data blocks it addresses directly (240 elements); its
super block addresses were always undefined. Chunks from index 244 on were
written to the file but never indexed, so they read back as fill values in
our reader and in libhdf5, without an error.
The writer now lays out data blocks and super blocks for any element
count as H5EA__hdr_init sizes them, pages data blocks larger than 1024
elements (page-init bits in the owning super block), leaves blocks with no
defined element unallocated, and records real header statistics
(max_idx_set is one past the highest defined index).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
read_vl_bytes cut each element to the reference's length field, which
counts sequence elements, not bytes: a VL int32 [1, 2, 3] came back as
3 bytes. Return the whole global-heap object, which is element count x
base size bytes. No in-tree caller depended on the old behaviour.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
read_i64/read_u64/read_i32/read_f64/read_f32 refused enumeration
datatypes, including h5py's bool (an enum of int8), with a type
mismatch. Read them as their base type's integer values, the way array
datatypes already read through theirs.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Every 2-byte float was decoded as IEEE half, so bfloat16 (HDF5 2.0's
H5T_FLOAT_BFLOAT16*, or any custom 8-bit-exponent type) read wrong:
1.5 as 1.9375, +inf as NaN. 1-byte FP8 floats were refused.
Read the exponent/mantissa location and size and the bias from the
datatype message: IEEE half/single/double keep their existing paths
(half still through clawhdf5_format::float16), any other IEEE-style
layout up to 64 bits whose values fit f64 (bfloat16, FP8 E4M3/E5M2, ...)
is decoded generically, and the bulk-copy and zero-copy fast paths now
require the IEEE layout rather than just the size. Datatypes with fields
that describe no float still fall back to IEEE by size; layouts that
cannot be represented in f64 (x87 80-bit, binary128) remain an error.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Reading wider or differently-signed integers kept the low bits: i64
2^40+5 read as i32 was 5, u64::MAX read as i64 was -1, and -1 read as
u64 was 4294967295. u32 data read as i32 also took the bulk-copy fast
path meant for i32. Saturate at the target range like libhdf5's hard
conversions (a negative value read as unsigned is 0), and keep the i32
fast path to signed data.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
read_i32/read_i64/read_u64 on a floating-point dataset reinterpreted the
IEEE bits (1.5 read as i64 was 4609434218613702656). Convert like
libhdf5's hard conversions instead: truncate toward zero and saturate at
the target range; NaN reads as 0.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Layout message v4 flag bit 0 (H5D_CHUNK_DONT_FILTER_PARTIAL_CHUNKS, set
with H5Pset_chunk_opts) makes libhdf5 store every chunk that extends past
the dataset's extent without the filter pipeline, while its filter mask
still reads 0. The parser ignored the flag, so readers tried to inflate
raw bytes: libhdf5's own h5fc_edge_v3.h5 failed with "deflate: ...
unknown compression method".
DataLayout::Chunked gains dont_filter_partial_edge_chunks (always false
for v3), and list_chunks — the one place every read path gets its chunk
list from — marks such partial chunks as having skipped every filter, so
the full, cached, indexed, parallel and selection readers all copy them
as-is. chunked_write.rs gets `..` in one exhaustive test pattern for the
new field.
Regression: libhdf5_edge_chunk_fixture_reads (h5fc_edge_v3.h5 from the
HDF5 tools test files, committed as a 2.5 KB fixture), and
h5py_unfiltered_partial_edge_chunks_read (the flag set through h5py's
bundled libhdf5 via ctypes, as h5py has no binding for it: fixed array,
extensible array and B-tree v2 indexes, 1-D and 2-D, plus a hyperslab
of the last chunk), and v4_chunked_dont_filter_partial_edge_chunks_flag.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The Fixed Array writer always packed every element into one data block
behind one checksum. Past 2^10 elements libhdf5 (and our reader) expect a
paged block: a page-init bitmap after the prefix, then one checksummed page
per 1024 elements. Any dataset with more than 1024 chunks and no unlimited
dimension failed with "incorrect metadata checksum" in h5py, h5dump and
our own reader.
build_fixed_array_at now takes one Option<WrittenChunk> per array slot so
later fixes can leave unallocated slots.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Datatype::serialize returned an empty message for these four classes, so
any dataset or attribute of them (including a Raw attribute copied from
another file) was unreadable by libhdf5 ("ran off end of input buffer
while decoding"). They now encode exactly as libhdf5 does: legacy object
and region references as datatype version 1, H5T_STD_REF kinds as version
4 with their encoding version, opaque tags NUL-padded to 8 bytes.
Parsing an opaque tag now stops at its first NUL, so libhdf5's padding
no longer becomes part of the tag. Datatype::check_encodable rejects
what has no encoding (an opaque tag over 248 bytes); FileWriter::finish
calls it for every dataset and attribute type.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Every decode stage was capped at the chunk's decoded size. That holds
only when every filter ahead of the codec preserves size; Fletcher32
does not (it appends a 4-byte checksum), so a pipeline with Fletcher32
before deflate (NetCDF-4's fletcher32 -> shuffle -> deflate ordering,
h5repack's "all filters") failed with "deflate: output exceeds size
limit" on every chunk.
decompress_chunk_masked now computes each stage's bound by running the
chunk size forward through the filters that precede it in write order
(and that the chunk's mask did not skip): shuffle keeps the size,
Fletcher32 adds 4, any codec adds at most n/8 + 64. The cap is still a
small constant factor of the chunk, so a decompression bomb is rejected
as before (tested).
Shuffle also had to learn libhdf5's handling of a length that is not a
whole number of elements (chunk + checksum): shuffle the whole elements
and leave the trailing bytes in place, in both directions. It used to
refuse such data.
Regression: h5py_fletcher32_before_deflate_reads (fletcher->shuffle->
gzip, fletcher->gzip, shuffle->fletcher->gzip, and a 2-D i32 grid),
fletcher32_ahead_of_deflate_stays_bounded and
shuffle_leaves_a_partial_trailing_element_in_place.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Pcodec chunks were written as filter 32023, which the HDF Group registry
assigns to Granular BitRound (GBR). Pcodec has no registered ID (checked
2026-09-25 against hdf5_plugins/docs/RegisteredFilterPlugins.md, which
ends at 32033 with no pcodec entry). GBR's decode is a pass-through, so
libhdf5 with that plugin loaded would have returned the compressed bytes
as the dataset's values.
Write pcodec as 480, from the registry's testing/private range (256-511),
named "pcodec (clawhdf5 private)", and document it as non-interoperable:
only clawhdf5 with the `pcodec` feature reads it. Chunks under 32023 are
still read as pcodec when the filter is named exactly "pcodec" (what
clawhdf5 <= 2.7.0 wrote); any other 32023 is UnsupportedFilter.
Test: pcodec_uses_private_id_and_reads_legacy_32023.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
A chunk's filter mask has one bit per pipeline filter; bit i set means
filter i was not applied to that chunk (an optional filter that
declined, or a direct chunk write). Every read path treated any nonzero
mask as "no filters applied" and returned the stored bytes, so a chunk
that skipped only gzip in a shuffle+gzip pipeline came back still
shuffled (h5py write_direct_chunk with filter_mask=0b10: 8 of 32 values
wrong).
decompress_chunk_masked undoes the filters the mask leaves set and skips
the rest; an unsupported filter is no longer an error when the chunk
skipped it. The full, cached, sweep, indexed, parallel and selection
(partial_read) paths all use it, and a chunk is copied straight from the
file only when every filter was skipped. decompress_chunk is the mask-0
case.
Regression: h5py_partial_filter_mask_skips_only_masked_filters (1-D
shuffle+gzip with masks 0, 0b10 and 0b11; 2-D with 0b01; full and
hyperslab reads) and filter_mask_skips_only_the_masked_filters.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Filter 32015 chunks were written with the streaming encoder
(zstd::encode_all), whose frames carry no content size. The registered
HDF5 Zstandard filter (H5Zzstd.c, libhdf5 + hdf5plugin) sizes its output
from ZSTD_getFrameContentSize and fails on such frames, so h5py could not
read our zstd datasets ("filter returned failure during read"). Compress
with the one-shot API, which records the size.
Tests: zstd_frames_record_content_size (content size was None before),
hdf5plugin_reads_our_zstd (ignored interop test; failed before).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Both indexes place each chunk at a linear index computed from the
dataset's maximum dimensions (libhdf5's max_down_chunks), and the
Extensible Array first swizzles its unlimited dimension to the slowest
position. We linearised by the current dimensions, so any dataset whose
shape was smaller than its maxshape, or whose unlimited dimension was not
the first, read back scrambled without an error: h5py libver="latest"
files with maxshape (10, None) or (20, 10), and the libhdf5 test files
h5fc_ext*.h5 and test_ld.h5.
The linearisation now lives in chunk_grid (shared with the writers), and
slots beyond the current extent are ignored as the library does.
read_fixed_array_chunks / read_extensible_array_chunks take the
dataspace's max dimensions.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
A v2 object header message has a 2-byte size field. The writer truncated
larger sizes to 16 bits, so an attribute over ~64 KiB (or a compact
dataset of 65532-65535 bytes, whose layout message adds 4 bytes) produced
a file libhdf5 rejects ("message of unshareable class flagged as
shareable", "bad flag combination").
ObjectHeaderWriter::serialize now returns a Result and fails on any message
over MAX_MESSAGE_SIZE; FileWriter::finish propagates it. Compact storage
falls back to contiguous above 65531 bytes, the real limit. Dense storage
for large attributes remains future work.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Filter 32004 chunks were framed as a 4-byte little-endian size plus one
LZ4 block. That is not the registered HDF5 LZ4 format (H5Zlz4.c: 8-byte
big-endian total size, 4-byte big-endian block size, then per block a
4-byte big-endian compressed length and the block, stored raw when the
length equals the block size), so libhdf5 + hdf5plugin could not read
our LZ4 datasets and we could not read theirs (h5ex_d_lz4.h5:
"lz4: 0 is not a valid match offset").
Write the registered format (cd_values[0] is honoured as the block size,
default 1 GiB like the plugin) and read it, multi-block and raw blocks
included. Chunks in the old framing stay readable: an HDF5 chunk is under
4 GiB, so a registered chunk always starts with four zero bytes and is at
least 12 bytes long, while an old one starts with four zero bytes only
when empty (5 bytes).
Tests: lz4_reads_registered_hdf5_format (chunk of the HDF Group's
h5ex_d_lz4.h5, block size 3), lz4_writes_registered_hdf5_format,
lz4_reads_legacy_clawhdf5_format, and hdf5plugin_reads_our_lz4 (ignored
interop test; failed before with "filter returned failure during read").
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
A type-1 (raw data chunk) B-tree key holds the chunk size, the filter
mask and one offset per dimension, and those offsets are always 8 bytes:
they are dataset coordinates, not file addresses. The reader used the
superblock's size-of-offsets for them, so in a file with 4-byte offsets
every key was misparsed. Unfiltered chunked datasets read as zeros (with
stray bytes where a misread address landed on data) and filtered ones
failed with "deflate: truncated stream".
Only the sibling and child addresses follow size-of-offsets now. The
unit-test B-tree builder wrote keys the same wrong way, which is why its
tests passed; it now matches the format.
Regression: h5py_four_byte_offsets_chunked_reads (h5py, set_sizes(4, 4)
and (4, 8); 1-D and 2-D, unfiltered and gzip) and the unit test
collect_chunks_with_four_byte_addresses.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
An 82-byte fuzz_btree_v2 crash input from 2026-09-20 was left untracked
in fuzz/artifacts. Replayed today it runs cleanly: the depth cap and
record budget added to B-tree v2 traversal that day fixed it. It is now
in the committed fuzz corpus, and a robustness test replays the fuzz
target's exact code path on it so a regression fails CI rather than
waiting for someone to run the fuzzer.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The previous commit said clawhdf5 has no integration at all. ClawBrainHub
(clawverse/clawbrainhub) does use it: cbh-core reads and writes .brain
files through the facade, cbh-scanner uses the facade, and cbh-cli uses
clawhdf5_agent::bm25::BM25Index, all via path dependencies on this repo.
Checked on 2026-09-25 against main: it builds on its pinned toolchain and
its 204 tests pass. CLAUDE.md now records that, and that path
dependencies mean API changes here reach it directly.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
CLAUDE.md said ZeroClaw "imports this as a Cargo feature (clawhdf5
feature flag)" and uses clawhdf5 as its memory backend; the agent crate
called itself the "ZeroClaw agent memory HDF5 backend"; the migrator
claimed to read "the ZeroClaw layout". Checked on 2026-09-25 against
ZeroClaw v0.8.5 (its latest release), the osobh/zeroclaw fork (on
v0.8.5) and both histories back to February 2026:
- no `clawhdf5` feature, dependency or memory backend has ever existed
in ZeroClaw; its backends are sqlite, lucid, postgres, qdrant,
markdown and none, behind its own `Memory` trait;
- ZeroClaw's SQLite schema is a single `memories` table (id, key,
content, category, embedding, created_at, updated_at); the
migrator's memory_chunks/sessions/entities/relations layout never
existed in ZeroClaw, so it cannot read a ZeroClaw database.
Decision: withdraw the claims (as with OpenClaw); clawhdf5 is a
standalone library with no framework integration. The migrator's
default layout is documented as its own. ZEROCLAW_VERSION keeps its name
and value (it is the persisted `edgehdf5_version` writer tag) with a
doc comment saying it is unrelated to ZeroClaw.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The docs described a "drop-in" OpenClaw memory backend enabled with
`memory.backend = "clawhdf5"`. Checked against OpenClaw's source and
docs (v2026.2.26 through v2026.9.6): that config was never valid —
v2026.2-v2026.7 accepted only "builtin"/"qmd" and rejected unknown
keys, so a Gateway given it refuses to start, and v2026.8.1 (OpenClaw
2.0) removed the key. No plugin was ever built (no manifest, no
registration, no tools), nothing was tested against OpenClaw, the
linked github.com/redclawsystems/openclaw is a 404, and
@redclaw/clawhdf5 was never published.
Decision (2026-09-25): not pursuing an OpenClaw plugin for now; ZeroClaw
is the integration target.
- Remove openclaw-integration.md, openclaw-config.md and
migration-guide.md; add docs/openclaw.md: the status, what a memory
plugin needs against v2026.9.6 (plugins.slots.memory, manifest with
kind "memory", registerMemoryCapability / MemorySearchManager,
prebuilt native packages), and what this repo has as building blocks.
- README, QUICKSTART, USE_CASES, ROADMAP (Track 7 withdrawn), CLAUDE.md
and the `openclaw` module docs describe ClawhdfBackend as what it is:
a Markdown-oriented library backend, not an OpenClaw plugin. The
QUICKSTART example is corrected (the old one called a three-argument
create that does not exist) and states its limits.
- packages/clawhdf5-node: marked unpublished and broken, "private": true
so it cannot be published by accident; its bugs (snake_case vs
camelCase fields, wrong addon path, no way to store an embedding,
wrong WAL name) are recorded in docs/known-issues.md.
- Two broken rustdoc links fixed along the way.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Makes the README's "cryptographically verifiable memory" true.
With HDF5Memory::set_signing_key(key), every checkpoint stores a signed
manifest of the store: a SHA-256 per memory record (text, embedding as
stored, channel, timestamp, session, tags, deleted flag, activation) in
a Merkle tree, plus hashes of the settings (and WAL mark), sessions and
knowledge graph. The signature, public key and manifest hashes go in
/meta; the per-record hashes in /integrity/record_hashes, so
HDF5Memory::verify(path, &public_key) can say which records changed, not
just that something did. A forged manifest fails the signature.
Decisions, as agreed:
- the key is set on the open store and never persisted;
- a signed store refuses to checkpoint without its key
(MemoryError::SigningKeyRequired); remove_signature() is the
deliberate way back to unsigned;
- checkpoints only: saves still in the WAL are not covered, and verify
reports how many there are.
The hashes cover exactly what the file persists, in the form the loader
returns it (strings lose trailing NULs; an empty WAL mark is not
written), so untouched stores verify across any number of reopen and
checkpoint cycles. MemoryError becomes #[non_exhaustive] (it already
gains variants in this unreleased version).
CLI: keygen (owner-only key file), --signing-key / CLAWHDF5_SIGNING_KEY
on writing commands (create signs immediately), verify --public-key
(JSON; exit 2 if not valid), `signed` in create/stats output.
Tests: reopen/checkpoint cycles with awkward strings (f16 and f32),
refusal without the key, wrong and rotated keys, eight kinds of edit
each detected and located, a forged manifest, unsigned stores, NULs in
text, and an edit made in place with h5py that verify pinpoints.
Cost on tank (search_harness --signing-study --full, 3 runs): ~20% of a
checkpoint (+9 ms at 10K, +89-112 ms at 100K), verify 18.6 ms / 247 ms,
32 bytes per record in the file. New deps ed25519-dalek, sha2,
rand_core: pure Rust, the no-C check passes.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
It enabled nothing — the agent layer is always built — yet the README,
QUICKSTART and USE_CASES told people to pass it. Removed, with those
snippets fixed: they now depend on the git repository (nothing is on
crates.io, so `version = "2.0"` never resolved) and USE_CASES no longer
presents the `float16` feature as half-precision storage (that is
MemoryConfig::float16, on by default for new stores).
Breaking for anyone passing `features = ["agent"]`: drop it.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
consolidation_efficiency never finished: stopped after 19 minutes on one
core while building its 100K case. Not the consolidation cycle (linear:
17 us at 100 records, 2.16 ms at 10K) but the setup — every add_memory
scores the new record's novelty against the whole working tier, the
benchmark lets that tier reach 50K, and each comparison recomputed both
norms: ~5e9 comparisons of three passes each.
ImportanceScorer::score_surprise now computes the new record's norm
once, takes each comparison in one fused, 8-lane pass (dot product and
the other norm together), and splits a working tier of 4096+ records
across threads with the `parallel` feature. Same results: tested against
the old cosine formula, including shorter, empty and zero vectors and
the parallel path. The work stays quadratic in the working-tier size by
design; with regular consolidation the tier stays near
working_capacity (100) and inserts are cheap.
The complete run takes 8 min 10 s on tank and fills in the 100K cycle
row (46.66 ms) and the memory-reduction table, which had never been
published. The binary no longer prints a record-count ratio as a
"BM25 Speedup" (never measured; Part 1 measures search latency) or
claims sub-linear cycle scaling (its own numbers grow slightly faster
than linearly).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Every undated or pre-September section re-run on one machine on one day
(tank, AMD Ryzen 7 7800X3D, 2026-09-24, commit 5c8323c), 24 commands run
serially with the load average checked before each, with the command
recorded for each section. A separate check traced every changed number
back to the raw output; its corrections are applied (e.g. the on-disk
~820 B/record is float16 plus always-deflated text on a synthetic corpus
of 40 distinct texts, not float16 alone).
Two apparent regressions were isolated rather than published:
- knowledge-graph traversal: a real bug, fixed in the previous commit;
- the write path: v2.3.0 built and run on the same machine measures the
same as today, so the old 18 us / 6.17 ms figures (undated, other
hardware) are not reproducible; float16 adds ~2 us per save and the
int8 index nothing (both isolated by switching the bench's config).
Also:
- new multimodal_bench: cross-modal search at 1K/10K records, which the
README claimed but nothing measured;
- footprint_bench reports whether it built float16 or f32 stores and
takes --f32 (it kept printing "f32" after the default changed);
- README: performance tables, the "Why" table figures and the SQLite
migration section (from the previous migrate commit);
- CHANGELOG for this branch.
Not re-run: consolidation_efficiency's 100K row and its memory-reduction
part (stopped for time), and cross_platform.sh.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
bfs_neighbors and spreading_activation built an adjacency index over the
whole graph on every call (1efd82c), so a 2-hop BFS over 1K entities
paid to index every entity and relation first: 155 us, 6.5x the 24 us
the README quoted. Found by the dated benchmark re-run.
The index is now cached on KnowledgeCache and checked against a
fingerprint of the graph on each use — one pass over entity ids and
relation endpoints, no allocation — so any change, including direct
edits of the public entities/relations Vecs (schema.rs's load path
pushes to them), still triggers a rebuild. A test edits the graph
directly in every way (push, in-place rewire, pop + push at equal
length) between traversals.
tank, 2026-09-24: BFS 1K entities 155.1 -> 23.1 us, 100 entities
17.5 -> 5.23 us, spreading activation 100 22.8 -> 10.1 us.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
clawhdf5-migrate wrote a layout of its own (/chunks, /sessions,
/entities, /relations, root attributes, no /meta or schema_version) that
HDF5Memory::open rejects, so a "migrated" SQLite database could not be
used as agent memory — contrary to the README.
It now writes through the agent's own API (HDF5Memory::create/open,
save_batch, the session cache and the knowledge graph), so there is no
second copy of the schema:
- sessions and entities/relations carry over; deleted rows become
deleted records (or are left out with --skip-deleted);
- embeddings follow the library default (float16), --f32 opts out and
--float16 is a hidden no-op, as in clawhdf5-cli; the `half`-based
conversion is gone;
- every source row is checked before the output is created: a wrong
embedding length, an empty embedding, a dimension that differs from
an existing store's, or a float16 value beyond +-65504 is an error
naming the chunk id, and an existing store is left untouched;
- --incremental opens the existing store, adds only rows it does not
hold (matched by content) and follows the source's deleted flags;
- a source with no memory rows needs --embedding-dim;
- validation reads the result back with HDF5Memory::open_read_only,
compares every field (embeddings bit for bit, round_to_f16 of the
source for float16) and checks a migrated record is found by search.
clawhdf5-agent gains HDF5Memory::sessions()/sessions_mut(),
HDF5Memory::delete_batch (one save, all-or-nothing, no auto-compact),
SessionCache::add_at, and re-exports SessionCache/SessionEntry.
The old layout's per-dataset SHA-256 provenance attributes have no place
in the agent schema and are gone. An adversarial review found two
blockers (silent truncation of long embeddings; an --incremental
dimension check that could never fire) and four majors (a failed run
wiping the existing store, dim-0 stores, deleted-flag drift); all are
fixed with regression tests. 42 migrate tests, incl. h5py opening a
migrated store.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
MemoryConfig::float16 now defaults to true for new stores, on
measurement: on the full LongMemEval haystack with real MiniLM
embeddings every retrieval metric matched f32 (previous commit), and at
100K the file is 48% smaller with faster checkpoints and opens.
Existing stores are unaffected: every agent store has recorded
`float16 = false` in /meta and keeps it. A test opens the v2.5.0
fixture, saves and checkpoints, and checks the embeddings are still f32
with the old rows bit-identical; another checks a new store is float16.
CLI: `create --f32` opts out; like `--f32-index` it only ever switches
the default off. `--float16` is still accepted and now a no-op.
Values beyond +-65504 are refused, so f32 remains the choice for
unnormalised vectors — the upgrade note says so.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
`longmemeval_bench --float16` builds every per-question store with
MemoryConfig::float16, so the vector stage searches half-rounded
embeddings exactly as such a store holds them.
Full longmemeval_s (500 questions, ~494 turns each) with real
all-MiniLM-L6-v2 embeddings, f32 vs float16, on tank (CUDA): identical
at every Hit@k and MRR, turn and session level, in all eight modes —
bar RRF session MRR 0.9253 vs 0.9254 and one or two flips out of ~320
in which gold session ranks first. The f32 run reproduces the published
hybrid numbers exactly. The earlier float16 evidence was synthetic
clustered data only; this is the real-embedding check.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
`HDF5Memory::search(query_embedding, query_text, &SearchOptions)` is the
store's full search path. `SearchOptions::new(k)` is plain hybrid search
with the tuned default fusion; each further stage is opt-in:
- `with_sources([..])`: only records from these source channels. The
filter applies before ranking, so a filtered search still returns up
to k results, normalised over what it can return. The HNSW pool is
over-fetched in proportion to what the filter removes, and the allowed
records are scanned exactly whenever that costs fewer distance
evaluations than the index would (~pool x M) — and as the fallback if
the pool comes back short. Keyword matches are filtered too.
- `with_rerank(ReRankConfig)` re-ranks a max(3k, 10) candidate pool by
relevance, recency, source authority and activation;
`with_confidence(ConfidenceConfig)` drops low-confidence results;
`at_time(now)` pins the recency clock.
These were reachable only through the OpenClaw backend, which is now
`search` with both on. Its Hebbian boost now goes to the k results it
returns rather than the whole 3k candidate pool. `hybrid_search` and
`hybrid_search_with` are wrappers and unchanged (tested bit for bit).
Measured on tank (search_harness --options-study --full, 3 runs): at
100K every filter — 50%, 10%, 1% of the store, and records far from the
query — returns the exact filtered top 10, and none is slower than an
unfiltered search (1%: 2.3 ms vs 4.6 ms). Re-rank + confidence costs
about 3%. A first version decided between index and exact scan by pool
size vs store size; it measured 0.976 recall at 12.3 ms on the
far-from-query filter, which is why the rule compares costs instead.
Tests: tests/search_options.rs (filter correctness and full pages via
both paths, far-from-query fallback, edge cases, equality with
hybrid_search_with, re-rank recency, confidence, boost scope).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The setting was persisted in /meta and otherwise ignored: embeddings
were always written as f32. It now does what it says.
clawhdf5-format:
- `DatasetBuilder::with_f16_data` writes IEEE binary16 (numpy float16),
rounding to nearest-even, and `make_f16_type`.
- `clawhdf5_format::float16` holds the f32 <-> f16 conversions, the one
implementation the writer, the reader and the agent all use. Checked
against the `half` crate on 16.7M f32 values and round-trips all 65536
half values; the h5py interop tests confirm the rounding matches
numpy's bit for bit (4020 values incl. ties, subnormals, overflow).
- Reading little-endian float16 as f32 has a fast path.
clawhdf5-agent:
- A float16 store writes /memory/embeddings as half precision, and
`MemoryCache::half_precision` rounds each embedding as it enters the
cache (save, update, WAL replay, and on load of a store still f32 on
disk), so memory and file agree bit for bit and a store searches the
same before and after a reopen (tested).
- Values beyond +-65504 are refused with the new
`MemoryError::InvalidEntry` rather than stored as infinity, on every
save path; batches are all or nothing, and a rejected ephemeral entry
stays in the ephemeral tier. Breaking for exhaustive matches.
- CLI: `create --float16`. Off by default.
Measured on tank, 384-dim, six runs alternating order, medians
(search_harness --float16-study --full): at 100K the file goes from
154.0 to 80.8 MiB (-48%), checkpoint 752 -> 512 ms, open 300 -> 252 ms;
vector recall@10 against an exact scan and hybrid_search latency do not
change. At 10K open is 3 ms slower. Also a test that h5py opens a whole
agent store, f32 and float16, and decodes every dataset.
Docs: README, BENCHMARKS.md ("float16 embedding storage"), CHANGELOG
(including the h5py interop fixes in the previous commit), CLAUDE.md.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Two write-side bugs, both present in every release (the first at least
since v2.1.0), made libhdf5 refuse files written by clawhdf5. Our own
reader ignores both fields, and the interop suites only ever wrote f64
from our side, so nothing here caught them.
- Every f32 dataset: "sign bit position out of bounds". The float
datatype encoder hard-coded the sign bit's position (bits 8-15 of the
class bit field) to 63, which is right only for f64. It is now derived
from the type: bit_offset + bit_precision - 1. This covered every
agent store's embeddings, norms and activation weights.
- Every empty dataset: "invalid dataset size, likely file corruption".
It was written with a real address and size 0, which trips libhdf5's
`addr + size <= addr` overflow check. An empty contiguous dataset now
gets the undefined address, as libhdf5 writes it. This covered every
agent store without sessions or a knowledge graph.
Agent stores are rewritten in full at each checkpoint, so they become
readable at their next checkpoint on a fixed build; other files with f32
or empty datasets need rewriting. Both are recorded in
docs/known-issues.md.
Tests: the sign position byte for f32/f64, and h5py reading our f32
datasets (plain and chunked + deflate) bit for bit.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
rust-version = "1.92" in [workspace.package], inherited by every crate.
1.92 is the floor: wgpu (clawhdf5-gpu) requires it, and the whole
workspace, Python bindings included, checks cleanly on it. ci-test.sh
reads the version from Cargo.toml and checks the workspace on exactly
that toolchain, so the manifests and the README badge cannot drift from
what actually builds. The badge said 1.75, below edition 2024's own
floor.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The core crates (clawhdf5, -agent, -format, -io, -filters, -ann, -accel,
-netcdf4, -cli) now build no C by default: deflate defaults to zlib-rs,
a pure-Rust port of zlib-ng, and zlib-ng becomes the opt-in
`fast-deflate`, which overrides zlib-rs wherever it is enabled. A default
build no longer needs cmake or a C compiler.
Measured on tank, both builds run alternately, three rounds, medians:
zlib-rs is within 6% of zlib-ng on every HDF5 read and write (512x512
deflate-6 chunked write 1.458 vs 1.484 ms; 64 MB compressed read 64.4
vs 65.2 ms), and compressed output is byte-identical. Details in
BENCHMARKS.md, "Deflate backend".
Getting there took two fixes the first measurement exposed:
- zlib-rs needs `std` to detect SIMD at runtime. flate2 enables it via
its default `runtime_detection`, which `default-features = false` had
switched off, leaving zlib-rs 3.5x slower on inflate. The `zlib-rs`
features now enable it.
- Both deflate paths streamed through flate2's 32 KiB read/write
wrappers. They now hand the codec the whole chunk in one call, into a
buffer sized up front (~5% on chunked writes). This also fixes a
silent short read: the streaming reader returned a truncated stream's
bytes without an error; a truncated chunk is now DecompressionError.
In clawhdf5-filters, output longer than the stated size is now an
error rather than silently cut off.
CI: ci-test.sh lints and tests the zlib-ng path, and fails if a
C-building crate (*-sys, cc, cmake) enters a core crate's default
dependency tree. The arm64 job no longer installs cmake.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The README had fallen behind v2.3.0-v2.7.0, and parts of it were not
true. Checked every claim against the code and BENCHMARKS.md:
- Three of the six Quick Start snippets no longer compiled (Agent
Memory, Consolidation, OpenClaw); all six now do.
- Hybrid search was described as RRF throughout. The default has been
weighted 0.4/0.6 fusion since v2.5.0; re-ranking and confidence
rejection run only in the OpenClaw backend.
- The `float16` feature does not halve embedding storage (the store
always writes f32), `--features agent` enables nothing, "Source
Isolation" is not wired in, and nothing backs "billion-scale" IVF-PQ.
- "Cryptographically verifiable" overstated an unkeyed, session-scoped
FNV-1a ledger; "Zero C dependencies" was false while zlib-ng was the
default deflate backend.
- Stale numbers: tests (1,650 -> 1,868), Rust badge (1.75 is below
edition 2024's floor), 6.5 KB/record on disk (BENCHMARKS.md: 1.7 KB),
consolidation and hybrid-search latency, and a feature-flag table
broken by a paragraph pasted into it.
- The file schema, module table and crate map now match the code.
Adds a "What's new (v2.2 -> v2.7)" section for collaborators, leading
with the silent Extensible Array read bug fixed in v2.7.0. Footer links
point at git.redclaw.dev. CLAUDE.md: clawhdf5-migrate is the SQLite
migration tool, and MemoryConfig::compression is off by default.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Two jobs, which runners serve them, and the two constraints that kept
the x86 job failing on every push until today: no JavaScript actions
(`rust:latest` has no `node`, and GitHub is not reachable from every
runner) and `cmake` for libz-ng-sys. Also notes that the Docker Hub
`latest` tag for the runner is frozen at 0.6.1, so it is not a way to
stay current.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Read from the job logs of the first run with the arm64 job, rather than
guessed:
- The x86 `test` job has been failing on every push, in about three
seconds. It runs in `rust:latest` and starts with actions/checkout, a
JavaScript action, and `rust:latest` has no `node`: exit 127 before a
line of code was built. It is replaced with a plain git checkout (the
image has git), and actions/cache, JavaScript for the same reason, is
dropped.
- `test-arm64` got through checkout, toolchain, the aarch64 check and
clippy, then failed building libz-ng-sys — pulled in by
clawhdf5-format's default `fast-deflate` — because the host runner had
no cmake. The x86 image lacks it too, so the x86 job would have hit the
same wall one step later.
Both jobs now install cmake where they can (the Docker job and the x86
container run as root) and say plainly when they cannot (a host runner),
instead of failing inside a build script. vision-01 now has cmake.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The NEON kernels in clawhdf5-accel — `dot_i8` including its SDOT path,
and the f32 NEON kernels that predate it — are cfg'd out on x86, so the
existing job has never compiled, linted or tested a line of them. They
were verified once, by hand, on a Raspberry Pi 5.
`test-arm64` runs on `linux_arm64`, which two runners serve in different
ways: vision-01 executes steps on the host with Rust preinstalled, and
vision-02 executes them in docker.gitea.com/runner-images. The job is
written to work in both: no `container:`, no JavaScript actions (those
are fetched from GitHub, which not every runner reliably reaches), and
an explicit `+stable` toolchain so a host's default — vision-01's is a
January nightly — is neither relied on nor changed. Fetches retry, since
one runner's outbound network was seen failing intermittently.
It lints the accel crate and tests accel, ann and format. It reports
rather than requires the dot-product extension: on a core without it the
plain-NEON kernel is the one that runs, and the tests cover whichever is
present.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
quantized_index defaults to true for new stores — smaller and faster at
equal recall on every configuration measured. Existing stores keep
their setting, and stores predating it stay f32, guarded by a real
v2.5.0 store committed as a test fixture.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
`MemoryConfig::quantized_index` now defaults to `true`. It holds a
quarter of the index memory and, with the exact re-score, is faster at
equal recall on every configuration measured: 1.63x the queries per
second on x86-64 (AVX2) and 1.18x on a Raspberry Pi 5 (NEON SDOT), with
builds 1.8x and 2.3x faster. The one argument for keeping it off — that
int8 search was slower on ARM — did not survive being measured.
Existing stores do not change. A store written with v2.6.0 or later
keeps its persisted setting. One written before the setting existed has
no stored value, and it loads as `false` rather than as the new default,
so reopening it never changes how its index is held. That case is
guarded by a real store written with the v2.5.0 CLI, committed as
`tests/fixtures/store_v2_5_0.h5` (6.8 KB): the test asserts it reopens
with an f32 index and still searches, and it fails if the load default
is changed to `true`.
The CLI needed more than a new default. `create --quantized-index`
assigned its value straight into the config, so under the new default
every CLI-created store would have been forced back to f32 unless the
caller knew to ask for int8. It is replaced by `--f32-index`, which only
ever switches the default off; `--quantized-index` is still accepted,
hidden, as a no-op, and the two conflict.
The whole agent suite passes under the new default, including the
brute-force recall oracle, now running on int8 plus re-score without
being asked to.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
SDOT and plain-NEON kernels for dot_i8, tested bit-exact against scalar
on real ARM. At equal recall the quantised index is 1.18x f32 on a Pi 5
and builds 2.3x faster. Also corrects an unmeasured claim that it was
slower than f32 on ARM.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
On a Raspberry Pi 5 at N = 100 000 and equal recall (0.9940 vs 0.9945),
medians of three runs:
f32 33 413 ms build 6 164 QPS
int8 scalar 18 950 ms ~6 190 QPS (what v2.7.0 shipped)
int8 NEON ~17 000 ms 6 640 QPS
int8 SDOT 14 464 ms 7 267 QPS 1.18x f32, 2.3x build
The docs said quantised search stayed off by default because aarch64
"falls back to the scalar loop, where the original trade still
applies" — that it was ~13% slower than f32 there, as on x86. That was
extrapolated rather than measured, and it was wrong: x86's portable
baseline is SSE2 against hand-written AVX2 f32 kernels, but on aarch64
NEON is the baseline and the scalar loop vectorises well, so it already
matched f32. Corrected in BENCHMARKS.md, README.md and CLAUDE.md; the
released v2.7.0 changelog entry is left as it was and the correction is
recorded in a new one.
Labelled as Pi 5 figures throughout — a Pi's memory bandwidth and cache
are far below an M-series or flagship phone, so the ratios will move.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
`dot_i8` had an AVX2 kernel and a scalar fallback, so on aarch64 the
quantised HNSW index ran the scalar loop. It now dispatches to one of
two NEON kernels:
- `dot_i8_dotprod`: the ARMv8.2 dot-product instruction, `SDOT`, which
multiplies and accumulates sixteen i8 pairs into four i32 lanes per
instruction. Present on Cortex-A76 and later (Raspberry Pi 5, current
Android phones), Neoverse-N1 (Graviton2, Ampere Altra) and every Apple
Silicon generation. Issued as inline assembly because the `vdotq_s32`
intrinsic is still behind the unstable `stdarch_neon_dotprod` feature;
inline asm is stable on aarch64.
- `dot_i8`: plain NEON for cores without the extension — `vmull_s8`
widens to i16 (even -128 * -128 fits) and `vpadalq_s16` folds adjacent
pairs into i32 accumulators, so nothing overflows.
Selected at runtime with `is_aarch64_feature_detected!("dotprod")`.
Verified on a Raspberry Pi 5 (Cortex-A76, `asimddp` present), not just
compiled — the aarch64 code is cfg'd out on x86, so x86 CI never builds
or lints it:
- both kernels bit-exact against scalar at every length, tails and
extremes included. Each is tested directly rather than through
dispatch, because dispatch only takes one path on a given CPU: on the
Pi, testing through it alone would never have run the plain-NEON
fallback at all.
- mutation-checked: dropping the SDOT kernel's second accumulator fails
at length 32, and using the low half twice in the NEON kernel fails at
length 16 — the first lengths that exercise each.
- the ANN suite passes, including int8 recall against ground truth.
- clippy clean with -D warnings on aarch64.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Graph degree and both candidate-list sizes were constants, so recall
could not be traded against memory or query speed. Now MemoryConfig
fields, persisted with the store, defaulting to today's behaviour.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Graph degree and the build- and query-time candidate list sizes were
constants, so a deployment had no way to trade recall against memory or
query speed. They are now `MemoryConfig::hnsw_m`,
`hnsw_ef_construction` and `hnsw_ef_search`, persisted with the store
and defaulting to exactly the previous behaviour (16, 64, and a query
list that scales with `k`).
Two things the straightforward version would have got wrong:
`clawhdf5-ann` asserts a graph degree of at least 2, so a configured 0 —
from a file, or from a caller reading 0 as "use the default" — aborted
the process inside the index builder. The store clamps instead, and a
test covers it: removing the clamp makes that test panic rather than
fail.
`ef_search` and the candidate pool handed to score fusion were the same
number. Tying the pool to the new setting would mean lowering `ef` for
speed also narrows what fusion sees, quietly degrading hybrid results
through a knob that looks like it only costs time. They are now
independent.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
read_from_disk mapped the file and then copied the whole mapping for
File::from_bytes, which maps it itself. Store open is ~28% faster
(455 ms -> 327 ms at 100k x 384); peak memory is unchanged, because the
peak falls after the parse during the index build.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
`read_from_disk` memory-mapped the file and then copied the entire
mapping into a `Vec` to hand to `File::from_bytes` — but `File::open`
memory-maps it itself whenever the facade's `mmap` feature is on, which
it is by default. So every open mapped the file, memcpy'd all of it, and
parsed the copy.
Store open at 100k x 384: 455 ms -> 327 ms, about 28% faster (two runs
after the change, 326.8 and 328.1 ms).
Peak memory is unchanged, which is worth saying because the opposite is
the natural assumption. The footprint harness now tracks a high-water
mark next to the retained figure, and it shows the peak falling after
the parse, during the index build — so a buffer allocated and freed
inside the parse never reaches it. Confirmed rather than assumed:
holding a deliberate extra copy of the whole file across the parse
leaves the peak exactly where it was, which is also what proved the
instrument was working before trusting its answer.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Fixed and Extensible Array structures all carry a Jenkins checksum that
was ignored. A single flipped bit in a chunk address parses cleanly and
points inside the file, so without the check the reader returns another
chunk's bytes as data.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Every structure in both chunk indexes — header, index block, super
block, data block and each data block page — carries a Jenkins lookup3
checksum, and all of them were parsed past and ignored.
What that costs is not a warning but correct data. Flip one low bit of a
chunk address and the index still has the right shape, the address still
lands inside the file, and the reader returns whatever bytes now sit
there as that chunk's contents. Nothing else in the parse can tell.
Verified in both directions. The checksums accept files written by
HDF5 2.0 from 100 to 200 000 chunks — dense, sparse, gzip-filtered and
paged — which also confirms the block layouts byte for byte, since a
wrong offset would fail every file. And an interop test corrupts an
address to check the read fails instead of returning data: removing the
verification makes that test fail with "corruption produced data instead
of an error", which is what it is there to prove.
The first version of that test passed with verification disabled — it
corrupted a byte a structural check already rejected, so it proved
nothing. Worth recording, since a test that passes for the wrong reason
looks exactly like coverage.
Hand-built fixtures now stamp real checksums, as HDF5 writers do, and
the Extensible Array ones no longer describe the superseded layout.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The quantised HNSW index was measured against f32 with a scalar loop on
one side and clawhdf5-accel's AVX2 kernels on the other, so the ~13%
throughput cost attributed to quantisation was a missing kernel.
With clawhdf5_accel::dot_i8 in place, at equal recall the quantised
index answers 1.63x the queries per second and builds 1.8x faster,
holding a quarter of the vectors. Still off by default, now because the
kernel is AVX2-only and aarch64 falls back to scalar.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Measured with `clawhdf5_accel::dot_i8` in place, medians of three
alternating runs at N = 100 000 x 384, same binary:
build f32 3197 ms int8 1778 ms int8+re-score 1826 ms
ef = 64 f32 13 399 QPS @ 0.9945 int8+re-score 21 848 QPS @ 0.9940
So at equal recall the quantised index is 1.63x the queries per second
and 1.8x the build speed, holding a quarter of the vectors. The earlier
"~13% of QPS" figure compared a scalar int8 loop against hand-written
AVX2 f32 kernels and was measuring the missing kernel; it is kept in
BENCHMARKS.md with that explanation rather than quietly replaced.
Still off by default, now for portability rather than performance: the
kernel is AVX2-only and aarch64 falls back to scalar, where the original
trade applies. A NEON kernel would settle it.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The int8-quantised HNSW index compared vectors with a scalar loop that
the compiler vectorised for the x86-64 baseline (SSE2), while the f32
path it was measured against goes through `clawhdf5-accel` and runs
AVX2. So the ~13% throughput cost recorded for `quantized_index` was a
missing kernel rather than a property of int8.
`clawhdf5_accel::dot_i8` adds a scalar fallback and an AVX2 path:
sign-extend each 16-byte half to i16, then `madd_epi16`, which
multiplies and sums adjacent pairs straight into i32 lanes. It is
dispatched through the same detected backend as the f32 kernels, and
the index now calls it.
Integer arithmetic, so the SIMD path must agree with scalar bit for
bit — tested at lengths that are and are not multiples of the block,
and at the -128 extreme for overflow.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Two read-path bugs found by auditing the improvement plan against the
code, both reaching every release up to v2.6.0.
Datasets indexed by an Extensible Array — any dataset with one
unlimited dimension — returned data from the wrong chunks past their
first few dozen, silently. The only fixture in the suite had three
chunks, inside the inline limit, so no test had ever read a data block.
A crafted file could abort the process through unbounded B-tree v2
recursion, or exhaust memory through shared subtrees. Both now refuse
in under a millisecond, and the fuzz target that should have caught it
(it only fuzzed header parsing) now walks the tree.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The Extensible Array bug reached a release because no fixture had more
chunks than fit inline, so its data blocks were never read. The Fixed
Array index had the same blind spot: nothing exercised it above a
handful of chunks, and nothing reached the paged layout at all.
Checked at 100, 5 000 and 200 000 chunks plus a sparse dataset that
leaves whole pages uninitialised. It is correct throughout — it does
keep its page-init bitmap inside the data block, which is the
difference from the Extensible Array that made assuming otherwise a
bug. Adding the tests so that stays true.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
A dataset with exactly one unlimited dimension — the ordinary
append-only case — is indexed by an Extensible Array. Only its first
few chunk entries (4 by default) sit inline in the index block, and
everything past them was read with the wrong layout. In the default
shape the 37th chunk onward came back from the wrong place: a
400-chunk dataset returned 364 wrong values while reporting success,
and beyond about a thousand chunks the read failed outright. Silently
wrong data is the worse half of that.
It survived because the only Extensible Array fixture in the suite had
three chunks — inside the inline limit — so no test ever reached a data
block.
Four layout errors, each confirmed against files written by HDF5 2.0 and
against the library source rather than inferred:
- super block `u` owns 2^(u/2) data blocks, not 2^u;
- each holds 2^((u+1)/2) * data_blk_min_elmts elements — the two
quantities double every *other* level, a half step apart;
- a super block carries a block-offset field before its data block
addresses, which was not skipped;
- the page-init bitmap belongs to the super block, one bit per page
packed across all of its data blocks and read MSB-first, rather than
living inside the data block; a paged data block also ends its prefix
with a checksum before the first page.
Where the spec left room for doubt the file settled it: decoding a
paged block's elements and reading the chunk values they address
identifies the mapping exactly, and the bitmap's 68 set bits matched
the 34 data blocks x 2 pages that 200 000 elements need, which only
holds MSB-first.
New interop tests cross every boundary — 4, 37, 400, 5 000 and 200 000
chunks, the last with paged data blocks — plus sparse (uninitialised
pages taking fill values), gzip-filtered elements and a 2-D dataset.
All three fail against the old traversal.
Writing is untouched; this was a read-path bug.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The existing target only called `BTreeV2Header::parse`, so the
recursive walk behind it — where a node that is its own child overflowed
the stack — was never fuzzed at all. Parsing also requires a valid
Jenkins checksum, which random input essentially never produces, so
almost every input stopped at the first branch.
The target now walks the tree after a successful parse, and also builds
a header straight from the input bytes so the traversal is reachable
without forging a checksum.
Checked both ways: against the unfixed traversal libFuzzer finds the
stack overflow (ASan: stack-overflow), and against the fix that same
input executes in 0 ms and 34.7 million further runs produce no crash,
timeout or OOM.
Corpora and crash artifacts stay out of the repository; the two crafted
inputs are covered by unit tests instead.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Traversal recursed one frame per level with the depth taken from the
file (a u16), and followed child addresses without asking whether they
were shared. Two crafted inputs, both reproduced before fixing:
- A node listing itself as its own child, under a header claiming 65 535
levels, overflowed the stack and aborted the process — SIGABRT, not an
error a caller can handle — from under 100 bytes.
- Levels whose children all point at one shared node below reached it
fan-out^depth times: 29.5 million records in 8 s from ~5 KB, and one
more level would exhaust memory.
Depth is now capped at 64, as the fractal heap already was; no real tree
approaches it, since even at the minimum fan-out of two that is over
2^64 records. And traversal stops once it has produced more records
than the file has bytes to hold them — a valid tree stores each record
once in its own bytes, so this bounds shared subtrees without trusting
the header's own `total_records`. Both inputs now fail in under a
millisecond.
Every B-tree v2 user goes through this collector: dense attributes, v2
groups, shared messages and chunk indexes. To show the budget never
refuses a real file, a new interop test has HDF5 2.0 write a depth-2
chunk index with 40 000 records and reads back all 160 000 values; it
fails when the budget is deliberately made too tight.
Also corrects `BM25Index::search`, which claimed to use Block-Max WAND.
It scores exhaustively, and pruning would not help the store:
`hybrid_search` needs every score because fusion normalises over them.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The workflow already installed h5py into /opt/interop and set
CLAWHDF5_REQUIRE_INTEROP=1, but it reached the tests only by appending
that venv to $GITHUB_PATH, which Gitea's runner does not reliably
propagate into test subprocesses. If `python3` resolved to the system
interpreter instead, every interop suite would skip. Setting
CLAWHDF5_PYTHON outright removes the question: together with
REQUIRE_INTEROP the suites either run or the build goes red.
Verified both directions locally — with a venv the four suites run 94
tests green; with a bogus interpreter and REQUIRE_INTEROP=1 the facade
and netCDF4 suites fail 22 tests rather than skipping.
Also documents creating the local `.venv` that `ci-test.sh` detects.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Cuts a loaded store's memory from 2.72x to 1.74x the raw vectors at 100k
x 384 via MemoryConfig::quantized_index, with recall held at the f32
index's level by re-scoring candidates against the exact embeddings the
store already holds. Off by default: it trades ~13% of QPS for the
memory.
Also restores the Python interop suites, which had been skipping
silently on this machine because no interpreter has h5py and PEP 668
blocks installing it into the system one.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Every Python interop suite had stopped running on this machine: the h5py
writer round-trips, the facade suite, netCDF4 and the reference files.
`python3` is 3.14, nothing on the box has h5py, and PEP 668 refuses to
install it into a system interpreter at all — so the availability probes
all returned false and each suite skipped without failing.
A silent skip here is exactly how the v5 compound-datatype bug reached a
release, so the probes now read `CLAWHDF5_PYTHON` and `ci-test.sh` picks
up `.venv/bin/python` on its own. The detection sits at the top of the
script rather than beside the interop step, because the non-ignored
suites run in the earlier `cargo test` step and would otherwise still
miss it. `CLAWHDF5_REQUIRE_INTEROP=1` continues to turn a skip into a
failure.
Verified against a venv with h5py 3.16 / HDF5 2.0.0: 94 interop tests
across the four suites, all passing.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
`MemoryConfig::quantized_index` stores the HNSW index's own copy of the
embeddings as i8 rather than f32. At 100k x 384 that takes the index from
266 to 123 MiB and the whole reopened store from 399 to 256 MiB — 2.72x
to 1.74x the raw vectors, the largest remaining item in the footprint.
Quantised distances are approximate and `ef` cannot compensate, because
the loss is in the distances rather than in the graph: recall@10 tops out
at 0.967 against f32's 0.9995 and does not move between ef=128 and
ef=256. The store already holds the exact embeddings, though, so when the
index is quantised the query path re-scores the candidate pool against
them before fusion. That restores recall (0.9940 vs 0.9945 at ef=64) and
costs about 13% of QPS.
Off by default: it trades query speed for memory and which side is worth
more depends on the deployment. The flag is persisted in `/meta`, so a
reopened store does not silently revert to four times the index memory,
and the sidecar graph is rehydrated into the configured storage.
Also on the CLI as `create --quantized-index`.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The HNSW index keeps its own copy of every vector, which at 100K x 384
f32 is ~146 MiB — the largest single item in the 2.43x footprint now
that the agent stores embeddings once. `Storage::Int8` cuts that copy to
a quarter by scaling each row to i8.
The scale is per row, not global. Unit-length rows in d dimensions have
components around 1/sqrt(d), so a fixed [-1, 1] scale spends fewer than
12 of the 255 levels on a 128-dimensional vector; measured against an
exact ranking that gives 0.35 top-10 overlap. Scaling each row by its
own largest component uses the full range and brings it to 0.99.
Quantised distances still cost recall on their own, and `ef` does not
buy it back because the loss is in the distances rather than the graph:
at N=100K recall@10 tops out at 0.967 against f32's 0.9995. Re-scoring a
wider candidate pool against the exact vectors removes the gap
(0.9940 vs 0.9945 at ef=64) for ~13% of query throughput and ~16% of
build time. That is the intended use, so it is what the test asserts —
against ground truth, not against the f32 index, whose own mistakes a
re-scored search is entitled to get right.
Default is unchanged: `Storage::Float32`, chosen by every existing
constructor. Serialized indexes carry f32 vectors and no storage tag, so
a quantised index is rebuilt rather than loaded; `compact()` keeps the
storage it was given.
The harness grows `--int8` and `--rerank` axes, and reports the storage
in each table header.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
MemoryCache held every embedding in two places: a `Vec<Vec<f32>>` and a
flattened copy for the batched kernels, kept in lock-step on every push,
update and compaction. A store loaded from disk therefore carried the corpus
twice, plus one heap allocation per entry.
A new `cache::Embeddings` owns just the flat `[N x dim]` buffer and indexes
into it, so `embeddings[i]` still reads as a `&[f32]` row. The batch kernels
take a `VectorSet` (implemented for both `Embeddings` and `Vec<Vec<f32>>`)
instead of `&[Vec<f32>]`, so their callers and tests are unchanged. Loading no
longer unflattens what it just read.
100k 384-dim entries, reopened from disk: 505 -> 357 MiB, 3.44x -> 2.43x the
raw vectors. Recall (1.0000 at ef=64) and query latency are unchanged.
Rows are now always exactly `dim` long, shorter ones zero-padded. The old
representation allowed ragged rows, which silently misaligned the flattened
copy — every row after a wrong-length embedding — and `update` carried a
comment about falling back to a rebuild to avoid exactly that. It is now
unrepresentable. A record saved without an embedding holds a zero row and is
told apart by its norm, which is what `total_embeddings` now counts.
Measured with a counting allocator rather than RSS: freeing a structure
returns its pages to the allocator's pool, not the OS, so an RSS reading from
inside the process showed the two representations as identical.
Breaking: MemoryCache::embeddings changes type, embeddings_flat is replaced by
flat_embeddings(), rebuild_flat() is a deprecated no-op.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
reranker::rerank built its combined score from temporal decay, source
authority and Hebbian activation. RerankInput carried no relevance score, so
it could not have used one: re-ranking a candidate pool reordered it purely by
age and discarded the retriever's ordering. The OpenClaw backend re-ranks
every search, so that was its shipping behaviour.
Measured over the full LongMemEval haystack (500 questions, real MiniLM
embeddings), ordering by metadata alone costs 40.6pp of Hit@1 (11.0% vs 51.6%)
and two thirds of MRR (0.1829 vs 0.6430) — the results are the newest memories
in the pool rather than the ones answering the question.
RerankInput::relevance and ReRankConfig::relevance_weight (1.0 by default)
make relevance lead, with the metadata signals breaking near-ties. Retrieval
is preserved (Hit@1 52.0%, +0.4pp against no re-ranking; MRR -0.003) and
recency discrimination improves 6-7pp, from chance to ~52%.
A half-life sweep (1, 7, 30, 90 days) moves recency 1.4pp and MRR 0.003 —
inside the noise — because the temporal term is capped by its weight while
relevance gaps are larger. The 24-hour default is kept: there is no measured
reason to change it. The two ends of the trade-off are recorded in
BENCHMARKS.md rather than just the good news.
Breaking: RerankInput and ReRankConfig gained fields.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Two gaps in the LongMemEval harness, both of which had to close before any
recency feature could be judged.
The store was fed a synthetic counter (ts += 1.0 per turn) and the dataset's
own `haystack_dates` were ignored. Session order happened to be chronological,
so ordering was right, but the intervals were fiction — and exponential decay
is a function of the interval, so anything time-aware was being measured
against made-up ages. Dates are now parsed (civil-from-days, pinned against
reference values) and turns are spread over the minutes after their session
start; an unparseable date falls back to position so order still holds.
`newest_gold_first` measures what recall cannot. On a `knowledge-update`
question LongMemEval labels *both* the stale session and the one that
supersedes it as gold, so returning either scores as a hit even though only
one answers the question. The new metric asks whether the newest gold session
outranked the older ones. The current retriever scores 43-45% on it across
every mode — chance — which is the gap a temporal signal is supposed to close.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Bump all workspace crates, the node package and pyproject to 2.5.0, fold the
two unreleased sections together and add upgrade notes for the behaviour
changes.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The keyword stage had no stemming, so "training" and "trains" were unrelated
terms. bm25::TokenFilter::Stemmed strips common English inflections (plurals,
-ing/-ed, with consonant un-doubling) from documents and queries alike;
BM25Index::build_with and HDF5Memory::set_token_filter select it, and the
index records which filter built it so a stale one is rebuilt rather than
mixed.
Measured over the full LongMemEval haystack (500 questions, real MiniLM
embeddings) rather than adopted on principle — and it is a trade, not a win:
BM25 only Hit@1 53.8% Hit@5 75.0% Hit@10 81.6% MRR 0.6320
BM25 stemmed Hit@1 52.0% Hit@5 77.8% Hit@10 84.0% MRR 0.6320
Hybrid 0.4/0.6 Hit@1 51.6% Hit@5 81.4% Hit@10 87.8% MRR 0.6430
Hybrid stemmed Hit@1 50.2% Hit@5 81.4% Hit@10 88.2% MRR 0.6394
Conflation buys depth and costs the top rank: on BM25 alone MRR is unchanged
to four decimal places, the deeper gains exactly offsetting the rank-1 loss.
On the shipping hybrid configuration the vector stage already supplies most of
that recall, so the trade is narrower and slightly negative. Default stays
Plain; Stemmed is there for callers who want Hit@5/@10 over rank-1 precision.
The stemmer is deliberately conservative — it only strips inflections, and
only when the stem stays long enough to be meaningful, since an aggressive one
also conflates unrelated words. Tests pin both the pairs that must meet and
the pairs that must not.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Probing what QueryExpander::expand actually produces for LongMemEval questions
turned up two defects in the same helper.
`replace_word_case_insensitive` did a plain substring replace despite its
name, so acronym expansion fired inside ordinary words: "training" became
"trArtificial Intelligencening" ("ai") and "programming" became
"Pull Requestogramming" ("pr"). Nearly every acronym expansion of prose was
corrupt. Matching now requires word boundaries at both ends; real acronyms
(API, database) still expand in both directions.
The same helper searched `text.to_lowercase()` and then sliced `text` with the
offsets it found. That holds only while lowercasing preserves byte length, and
it does not — Turkish 'İ' is 2 bytes and lowercases to 3. Offsets after such a
character drifted, so output was silently corrupted ("İstanbul AI trip" lost a
character) or the slice landed inside a character or past the end and
panicked: `expand("İ AI")` was enough, from a plain query string. Matching now
walks the original string, comparing case-insensitively char by char, so
offsets are always valid.
Regression tests cover both, plus whole-word matching at string edges. The
morphological rules remain crude ("during" -> "dured"); that is a quality
limit, not a correctness bug, and is now documented as a reason to measure
before enabling expansion on a retrieval path.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Reciprocal rank fusion was implemented but reachable only as a free function
over a linear scan, so its merits had never been tested. With both running
over the same HNSW + BM25 candidates on the full LongMemEval haystack (500
questions, real MiniLM embeddings):
weighted 0.4/0.6 turn Hit@1 51.6% Hit@5 81.4% MRR 0.6430
RRF k=60 turn Hit@1 45.0% Hit@5 78.8% MRR 0.5967
RRF lands almost exactly where the old 0.7/0.3 weighting did, and for the same
reason: it combines the stages by rank with equal influence, but on this corpus
BM25 alone beats the vector stage by 17.8pp at Hit@1, so treating them as peers
costs rank-1 accuracy. RRF's advantage is robustness when the stages' scores
are not comparable and there is nothing to tune against; here there is, so the
weighted sum stays the default. Recorded in BENCHMARKS.md with the reasoning,
including that this is a property of the corpus rather than a defect in RRF.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
BENCHMARKS.md has recorded since the weight sweep that the 0.7/0.3 default is
strictly dominated by 0.4/0.6 over the full LongMemEval haystack, but the
shipping code never adopted it: unified_search and the OpenClaw backend both
passed 0.7/0.3. Re-running the sweep here (500 questions, real MiniLM
embeddings on a GPU) reproduces it — turn-level Hit@1 51.6% vs 44.2%, Hit@5
81.4% vs 79.2%, Hit@10 87.8% vs 85.8%, MRR 0.6430 vs 0.5856 — so both now use
hybrid::DEFAULT_FUSION, which is that operating point and carries the
reasoning. A unit test pins it.
Fusion is also selectable now. hybrid::Fusion is either Weighted { vector,
keyword } or Rrf { k }; hybrid::fuse applies either to one candidate list per
stage, and merge_vector_keyword / hybrid_search delegate to it, so the public
API is unchanged. New HDF5Memory::hybrid_search_with and
hybrid::hybrid_search_fused take a Fusion. Reciprocal rank fusion was
implemented but reachable only as a free function over a linear scan, so it
had never been compared with the weighted sum on equal terms; it is now a mode
in the LongMemEval bench (measurement to follow).
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
HDF5 1.12 revised the reference datatype (class 7) in datatype message version
4: reference types 2-4 are the new H5T_STD_REF object / dataset-region /
attribute references. Datatype::parse rejected them with
InvalidReferenceType, so any dataset of that type was unreadable.
h5py cannot write this type, which is why it had never been tested. A real
file was produced by calling the libhdf5 bundled in the h5py wheel through
ctypes (H5T_STD_REF_g, H5Rcreate_object, H5Dwrite); the 2 KB result is
committed as tests/fixtures/std_ref_hdf5_2_0.h5 with its generator,
gen_std_ref.py.
- ReferenceType gains Object2, DatasetRegion2 and Attribute, accepted only
from datatype version 4.
- read_object_references decodes Object2 elements: type(1) flags(1)
token_size(1) token, zero-padded to the element size; the token is the
target's object header address. A null reference decodes to the undefined
address; an external reference, a wrong type byte or a token that doesn't
fit is an error.
The fixture test follows both references and checks they resolve to the
objects they were created from.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Requesting a filter without chunk dimensions made the whole dataset a single
chunk. Any read, even one row, then decompresses everything, and a large
dataset cannot be decoded in parallel — which also made the new partial reads
pointless for such files.
auto_chunk_dims keeps datasets up to 1 MiB as one chunk (unchanged behaviour)
and splits larger ones by halving the dimensions in turn, so chunks keep
roughly the dataset's proportions, until a chunk is at most 1 MiB — h5py's
approach. An empty (unlimited, unwritten) dimension is treated as 1024. The
writer passes the element size through resolve_chunk_dims_for; the old
resolve_chunk_dims assumes 8-byte elements. Explicit with_chunks always wins.
Interop test: h5py reads an auto-chunked 13 MB deflate dataset, sees chunks
between 128 KiB and 1 MiB, and a small dataset still has one chunk.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
shuffle_decompress — on the read path of every compressed dataset, since
shuffle is applied automatically before compression — was the naive
`result[i * es + j] = data[j * n + i]`: a multiply and two bounds checks per
byte. It now interleaves fixed-width arrays of byte planes for element sizes
2/4/8/16 (bounds checks hoisted, vectorisable), with a chunked generic
fallback. The write-side shuffle was already optimised; this was the asymmetry
the survey flagged. Modest wall-clock effect now that decode is parallel
(chunked+deflate full read ~70 -> ~66 ms). Round-trip test over element sizes
1-24 and several lengths.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Same-moment A/B on a 64 MB f64 dataset: chunked+deflate 110 -> 69 ms, chunked
72 -> 60 ms, contiguous 56 -> 30 ms.
- read_chunked_data_cached — the path the facade uses — decompressed chunks
one at a time; only the uncached reader was parallel. Cache misses are now
decoded in bounded batches (128), in parallel with the `parallel` feature.
- Every chunk was pushed into the 16 MiB chunk cache, which a larger dataset
just churns (insert, evict moments later). Chunks are cached only when the
whole dataset fits (new ChunkCache::max_bytes).
- Unfiltered chunks went file -> Vec -> aligned cache buffer -> output. They
are copied straight from the file bytes.
- The facade's typed reads convert a contiguous dataset straight from the
borrowed file bytes instead of copying it into a Vec first.
- The native little-endian fast paths allocated vec![0; n] and then overwrote
it; they now fill an uninitialised buffer in one copy (native_le_to_vec).
alloc_output requests zeroed memory from the allocator instead of reserving
and filling.
The unit test that expected unfiltered chunks to land in the decompressed
cache now asserts the new design (index reused, cache not involved).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
With libver='latest', a chunked dataset with two or more unlimited dimensions
indexes its chunks with a v2 B-tree (layout v4, index type 5). Reading one
failed with "unsupported chunked layout version=4, index_type=Some(5)".
read_btree_v2_chunks decodes record types 10 (address + scaled offsets) and 11
(address, stored size, filter mask, scaled offsets). The width of the
stored-size field is taken from the record size the tree header declares
rather than re-deriving the library's formula. Scaled offsets are multiplied
back by the chunk dimensions with overflow checks.
The chunk-index dispatch existed four times (uncached, cached, sweep and
indexed readers). The three copies outside list_chunks now call it, so every
read path — and fill-value handling and partial reads — supports every index
type from one place.
h5py interop test: plain, gzip+shuffle, a 2500-chunk tree with internal nodes,
a sparse dataset with a fill value, and a strided hyperslab.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
read_raw_data_selection computed which chunks a selection intersects, threw
the answer away, decoded the entire dataset and picked elements out of it —
for contiguous layouts too. A 64x64 window of a 64 MB deflate dataset cost
105 ms, about half a full read; every selection cost the same whatever its
size.
New partial_read module: materialise only the selection's bounding box — the
overlapping rows of a contiguous dataset (straight from the file bytes) or the
overlapping chunks (only those are decompressed) — then run the existing
extractor over that buffer with the selection translated to the box origin, so
extraction semantics are exactly the full-read ones. It declines (falling back
to the old path) for All/None, compact/virtual/storage-less layouts, and boxes
covering more than half the dataset. That window now takes 0.39 ms, one row
2.7 ms, one column 5.2 ms.
Selections are validated against the dataset shape first. They were not: a
hyperslab past an edge came back padded with zeros and a point with an
out-of-range column wrapped into the next row, returning the wrong element
with no error. Now FormatError::SelectionOutOfBounds (also rank mismatch and
overlapping blocks); the facade's fill-aware path validates too.
Tests: equivalence against a reference extraction from a full read over 60
random hyperslabs/point lists per layout (contiguous, chunked, deflate) for
ranks 1-3. New read_harness bench binary with before/after in BENCHMARKS.md.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
`parallel` joins the agent's default features, so the HNSW bulk build uses the
thread pool: cold index build at 10K records 1152 -> ~380 ms in a same-moment
A/B (the graph is identical either way). Nothing else on the measured paths
changes — ingest, checkpoint, open and steady-state query times are the same
with the feature on or off. Adds rayon to the default dependency set; opt out
with `--no-default-features --features float16,hnsw`.
Harness: `--e2e-only` runs the end-to-end section without the index
benchmarks. Note for anyone comparing numbers: this machine's absolute timings
drifted ~1.5x over a long session, so only same-moment A/B runs are
comparable.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Profiling the build showed 90% of all distance evaluations are in back-link
pruning (40.8M of 44.9M at 10K): every overflow re-runs the diversity
heuristic pairwise over ~max_conn candidates.
The bulk build now inserts in batches: plan every node's neighbours against
the graph as it stood when the batch began (read-only, so plans are
independent), link, then prune each overflowing list once. A node gaining
several back-links in a batch is pruned once rather than once per link, so
this is faster even single-threaded (10K: 1676 -> 1074 ms). With `parallel`,
planning and pruning use rayon (10K: 388 ms; 100K: ~21 s -> 5.9 s on 16
cores). Batches start at one node and are capped at 1/16 of the linked graph
and 512 nodes; a node that raises the top layer gets a batch to itself. The
result is deterministic and identical with or without the feature (one code
path; test compares two builds byte for byte).
Parallelising within a single insert was tried first: 1.45x on 16 cores, tasks
too small. Incremental insert() stays sequential.
Recall on clustered data is unchanged or slightly better; uniform random data
dips slightly (10K, ef=64: 0.474 -> 0.444).
clawhdf5-agent's `parallel` feature now passes through to the index.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
search() collected ef candidates, then filtered out soft-deleted nodes, then
took k. When the records nearest a query had been deleted, every candidate was
a tombstone and the search returned fewer than k results — 39 of 40 queries in
the new test, which deletes each query's 40 nearest neighbours.
search_layer takes an optional skip mask: a skipped node is still pushed onto
the candidate queue (a tombstone is a valid waypoint) but never into the
result heap, so the ef result slots hold live nodes only. Build and insert
pass no mask. Recall and speed without deletions are unchanged.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Bump all workspace crates, the node package and pyproject to 2.4.0, finalize
the changelog and add upgrade notes for the search behaviour changes.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Fusion min-max normalises over every keyword match, so hybrid_search asked
BM25 for a ranked list of the whole corpus: a hash insert per posting, then a
sort of every match, then the merge sorted every candidate again to keep k.
- BM25Index::scores returns every match unsorted, accumulated in a dense array
(contributions are strictly positive, so zero means untouched). search() is
built on it with the bounded heap.
- merge_vector_keyword partitions out its top k (select_nth) and orders only
those, with the same score-then-id order.
- Both hybrid paths use scores().
Rankings are identical (equivalence tests for both changes). p50 0.24 -> 0.07
ms (1K), 2.1 -> 0.49 ms (10K), 23 -> 4.65 ms (100K).
The harness gains --fusion-study, which measured the alternative — capping the
keyword pool — and found it changes the top-10 for most queries (overlap
0.83-0.92, different #1 for 10-35%) for only a 2x saving. Not adopted.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
- Cosine distance was 1 - dot/(|a||b|), re-deriving both norms on every
evaluation in the innermost loop of build and search. The index now stores
unit vectors (prepared at build, insert, graph load and HDF5 load; the query
once per search) and uses 1 - dot. Zero vectors stay zero, giving distance 1
as before. Returned distances are unchanged.
- search_layer allocated a HashSet of visited nodes per call. It is now an
epoch-stamped u32 array in thread-local scratch, reused across calls, so
search(&self) stays shareable between threads.
- clawhdf5-accel caches the detected SIMD backend in a OnceLock.
Recall is identical. Build 2.75 -> 1.89 s (10K), ~38 -> 21 s (100K); QPS at
ef=64 22.7K -> 39K (10K), 10.4K -> 14K (100K).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
open() marked the HNSW index dirty, so the first search of every session
rebuilt it from scratch — 36 s at 100K records with the (better, slower)
heuristic build. First query after open is now 1.7 / 15 / 159 ms at
1K / 10K / 100K; what remains is the one-off keyword index build.
- clawhdf5-ann: HnswIndex::graph_to_bytes / from_graph_bytes serialize the
graph only (levels, tombstones, adjacency as u32, CRC32). The existing HDF5
serializer embeds a full copy of every vector, which would double a store
that already holds them. Loading validates everything — counts, levels vs
layer count, connection limits, every neighbour id and the layer it must
exist on — so a damaged graph, or a hostile one with a valid checksum, is an
error rather than an out-of-bounds walk during search.
- clawhdf5-agent: each checkpoint writes the graph to <store>.h5.ann (synced,
atomic, before the .h5) and records a fresh generation id in /meta. open()
loads the sidecar only if its generation matches that checkpoint; missing,
stale, damaged or mismatched sidecars are ignored and the index rebuilt.
Records appended through WAL replay join the loaded index incrementally; a
replayed Update or Tombstone invalidates it. snapshot() copies it. Only an
index that exactly mirrors the cache is saved; otherwise a stale sidecar is
removed.
- ensure_hnsw_fresh inserts records appended since the last sync instead of
rebuilding, so save_batch no longer marks the whole index dirty.
- CheckpointMeta { wal_applied, ann_generation } with *_with_meta build/write/
read functions; the *_with_mark ones delegate.
- Harness reports the one-off cold index build separately from the first query
after a reopen.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
hybrid_search rebuilt the BM25 index from scratch (re-tokenising every record)
and rewrote the whole .h5 file on every single query, so a query cost O(store
size) in both CPU and disk I/O. Steady-state p50 per the search harness:
5.5 -> 0.24 ms (1K), 49 -> 2.1 ms (10K), 884 -> 23 ms (100K).
- BM25Index is incremental: add_document / remove_document keep it exactly
equivalent to a fresh build over the same live documents (property test: 60
random op sequences compared against BM25Index::build after every step). IDF
moves to query time since it depends on the live document count. Top-k uses
a bounded heap, ties break by doc id (results were HashMap-ordered), and the
"WAND" code that computed a bound and then discarded it is removed.
- HDF5Memory keeps one index for its lifetime, built lazily. Appends are
picked up by ensure_bm25_fresh whatever path added them; delete and in-place
update report themselves; compaction drops the index. A test drives every
mutation and compares against a fresh build.
- A query no longer calls flush(). Activation boosts are marked dirty and
persisted by the next checkpoint, including a best-effort one on drop so a
search-only session keeps them (approved behaviour change). Activation
weights are capped at 16.0; they previously grew without bound.
The archived mission branch's BM25 cache was reviewed and not used: it was
invalidated by every write, so interleaved save/search still rebuilt per
query, and it changed the default fusion weights.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Neighbours were selected as the plain closest-M, for both a new node's links
and back-link pruning. On clustered data every link of a node inside a tight
cluster then goes to that same cluster, so clusters become islands a search
entering elsewhere can never reach: recall@10 was 0.87 / 0.67 / 0.31 at
1K / 10K / 100K (384-dim) and flat in ef. Uniform random data — all the
existing tests used — does not show it.
Implement the HNSW paper's Algorithm 4 with keepPrunedConnections: accept a
candidate only if it is closer to the node than to every neighbour already
accepted, then fill spare slots with the closest rejected ones. Recall@10 at
ef=64 is now 1.00 / 1.00 / 0.98 and rises with ef; uniform data improves
slightly. Build is ~3.5x slower at 10K (extra distance evaluations), to be
recovered by the distance-kernel work. The needless rayon fan-out over <=33
distances in prune_connections is gone.
Tests: a clustered-data recall test for bulk build and incremental insert
(scores 0.43 with the old selection), and a unit test of the selection rule.
Harness gains --uniform and --ann-only; before/after in BENCHMARKS.md.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
New clawhdf5-bench binary `search_harness`, the measurement baseline for the
search hot-path work. On deterministic clustered 384-dim data it reports HNSW
build time and, per ef, recall@10 against an exact scan, QPS and p50/p99; and
for HDF5Memory: ingest, checkpoint, open, first-query-after-open and
steady-state hybrid_search latency at 1K/10K (and 100K with --full). Optional
JSON output for tracking.
Baseline recorded in BENCHMARKS.md. It shows two problems: HNSW recall@10 does
not respond to ef and falls from 0.87 (1K) to 0.31 (100K) on clustered data,
and end-to-end hybrid_search is ~1000x slower than its vector stage because
every query rebuilds BM25 and rewrites the .h5 file.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Bump all workspace crates, the node package and pyproject to 2.3.0, finalize
the changelog and add upgrade notes for the behaviour changes.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
attrs() silently omitted any attribute whose datatype had no AttrValue variant
— including every Python bool, which h5py stores as an enum — plus complex,
compound and reference attributes, and cast unsigned 64-bit arrays to
I64Array so values above i64::MAX came back negative.
- numpy/h5py-style booleans (an enum of exactly FALSE=0 / TRUE=1 over an
integer base) decode as I64 / I64Array of 0/1.
- AttrValue::U64Array keeps unsigned arrays unsigned. Behaviour change: an
unsigned array attribute no longer arrives as I64Array; the netCDF-4 CF
helpers (_FillValue, valid_range) and the Python bindings handle it.
- AttrValue::Raw { datatype, shape, data } carries any other attribute
verbatim (also used when a value fails to decode as its declared type), so
the attribute list is always complete. Decodable with data_read against the
datatype; Python receives {"dtype", "shape", "data"}.
- Both new variants are writable, so attributes round-trip between files.
h5py interop tests cover reading 13 attribute kinds and h5py reading back a
compound and a u64 attribute written by clawhdf5.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
- tests/wal_properties.rs — deterministic generator, reproducible by seed:
everything appended is read back intact (300 cases), and after ANY damage
to the file (bit flips, truncation, inserted/deleted bytes, duplicated or
rotated regions, overwritten ranges; 1500 cases) reading never panics and
yields an exact prefix of what was written — the guarantee the chained CRC
exists to give. Opening for append then repairs the tail and a new entry
lands right behind the surviving prefix.
- tests/crash_recovery.rs — builds the on-disk images a process crash can
leave and reopens each against a model of what was acknowledged: an image
after every operation (random saves, in-place updates, checkpoints, small
wal_max_entries), the checkpoint window (new .h5 + not-yet-truncated WAL)
over several rounds, and the WAL torn at every byte length, which must
recover the checkpoint plus a prefix of the operations logged since.
- fuzz/fuzz_wal_replay — arbitrary bytes as a WAL: read and open-for-append
must not panic, and open() must not change what is replayable. Based on the
target from the clawmates mission branch (4aee2fa), with the repair
property added. The CI fuzz step now covers both fuzz crates.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
The VDS resolver joined the source file name stored in the HDF5 file straight
onto the opened file's directory. That name is untrusted: an absolute path
replaces the base directory outright and `..` components climb out of it, so
a crafted file could make the reader open any path the process can reach.
Only plain relative paths of normal components are accepted now; anything
else resolves to "source not found".
Co-Authored-By: Claude Fable 5.1 <[email protected]>
- Path resolution follows soft links in both old-style (symbol table, cache
type 2) and new-style (compact and dense Link message) groups: absolute and
relative targets, links to groups, links through links, with a depth limit
so a link cycle is NestingDepthExceeded rather than a hang. A dangling link
reports the target it could not find. Previously every soft link was
PathNotFound.
- An external link is FormatError::ExternalLinkUnsupported { filename,
object_path } instead of a misleading PathNotFound.
- Message 0x0007 (External Data Files) is now a known MessageType, and a
dataset carrying it is FormatError::ExternalDataFilesUnsupported. Such a
dataset has no data address in this file, so it would otherwise be read as
"never written" and answered with fill values — wrong data, no error.
- Dense link iteration is shared between hard-link listing and the new
symbolic-link lookup; entry listing behaviour is unchanged.
- h5py interop test for both libver settings.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
HDF5 allocates lazily: a chunk nobody wrote doesn't exist in the file, and a
dataset nobody wrote has no data address. Such regions must read as the
dataset's fill value. There was no Fill Value message parser at all, so:
- a sparse chunked dataset read its holes as zeros — silently wrong whenever
the fill value isn't zero (h5py `fillvalue=-1` came back as 0);
- a dataset that was created but never written failed with NoDataAllocated /
"no address for chunked layout" where h5py returns a filled array.
New clawhdf5_format::fill_value: parses Fill Value messages v1-v3 and the old
0x0004 message (validated against HDF5 2.0 output under default and latest
libver), builds a fully filled dataset when there is no storage, and writes the
fill value into exactly the chunk-grid cells absent from the chunk index —
never mistaking a stored zero for a hole, clipping edge chunks, any rank. It is
skipped entirely for the default (zero) fill value. The chunk index dispatch is
extracted from read_chunked_data into a reusable list_chunks.
The reader, lazy and mmap facades apply it on full reads; selection reads go
through a fill-aware full read when the fill value matters. h5py interop test
compares against h5py's own readback, including a sparse 2-D dataset and a
hyperslab straddling allocated and unallocated chunks.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
A dataset created from a committed (named) datatype stores only a shared-
message reference to it. The facade parsed those reference bytes as the
datatype itself, producing `Time { size: 0 }` and unreadable data, and an
attribute using a committed datatype was silently dropped.
- shared_message::parse_shared_ref had the encoding wrong: it skipped six
reserved bytes for version 2 (only version 1 has them) and had the version 3
types inverted (1 is the SOHM heap, 2 is "committed, in another object
header"). Verified against h5py 3.16 / HDF5 2.0, which writes
`02 02 <address>` under both default and latest libver bounds. Resolution
now dispatches on which field the reference carries.
- New shared_message::message_data resolves a header message through the
indirection; the reader, lazy and mmap facades use it for datatype,
dataspace and filter-pipeline messages.
- AttributeMessage honours the v2/v3 flags (bit 0 datatype shared, bit 1
dataspace shared) via the new parse_in_file, used everywhere file data is
available. Parsing a shared attribute without file access is now
FormatError::UnresolvedSharedMessage instead of a garbage datatype.
- h5py interop test covering both libver settings.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
- Dataset::filter_pipeline() (reader, lazy and mmap variants) swallowed parse
errors with `.ok()`, so a malformed pipeline message silently became "no
filters" and the still-compressed chunk bytes were returned as the data. It
now returns Result<Option<_>>; a present-but-unparseable pipeline is
Error::Format.
- FileBuilder::write used std::fs::write, which truncates the destination
first: a crash mid-write destroyed the existing file. It now writes a
sibling temp file, syncs it, renames it over the target and syncs the
directory, cleaning the temp file up on failure.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Dataspace and chunk dimensions are untrusted 64-bit fields, but the chunked
read paths computed `num_elements() as usize * elem_size` and
`chunk_dims.product() * elem_size` with plain arithmetic and fed the result to
`vec![0u8; n]`. A crafted file could wrap the product (under-sizing the output
buffer that chunks are then copied into) or request an allocation large enough
to abort the process.
- Dataspace::checked_num_elements, checked_byte_len, checked_chunk_byte_len
and alloc_output (try_reserve_exact) replace the plain products and
vec![0; n] at every chunked read site, plus the VDS and hyperslab paths.
Overflow and allocation failure are FormatError::Overflow.
- Dataspace::num_elements saturates instead of wrapping.
- A zero-element dataset returns early, which also keeps the stride products
in range when another dimension is huge.
- parallel_read.rs: the three `c_addr + size > len` bounds checks used a raw
add; they now use checked_add like the rest of the crate.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
test_hebbian_activation_boost failed intermittently. Root causes, all in the
query path:
- normalize_scores mapped a set of identical scores — including the
single-candidate case — to 0.0, so a lone perfect match contributed nothing
to the fused score. Identical positive scores now normalise to 1.0 (all
equally the best match); identical non-positive scores stay 0.0.
- merge_vector_keyword sorted a HashMap's entries by score alone and then
truncated, so which ties survived varied from run to run; hybrid_search had
the same problem in its final sort. Both now break ties by index.
- hybrid_search applied the Hebbian boost to every returned record, including
the zero-score filler that pads the list when fewer than k records match.
With random tie-breaking a filler record could collect as many boosts as the
real hit. Only records with a positive fused score are reinforced now.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
- ProvenanceStore::remap: compaction renumbers cache indices (which are the
provenance record ids) but nothing renumbered the ledger, so after any
compaction — including the automatic one in delete() — every surviving
record's hash was filed under a different record and the next
save_or_update raised a bogus High "integrity mismatch" alert.
- Pending anomaly alerts are capped (newest 1024 kept). Alerts never block a
save, and a session over its write limit alerts on every write, so a caller
that didn't drain them grew the queue without bound.
- WriteAnomalyDetector tracks at most 4096 sessions, forgetting the
least-active half on overflow instead of leaking one entry per session id
for the life of the process.
- snapshot() copies the pending WAL next to the .h5 copy, so a snapshot is
the store as it is now rather than as of the last checkpoint (it used to
silently omit up to wal_max_entries recent saves).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
- HDF5Memory::create/open take an exclusive advisory lock on <store>.h5.lock
(std File::try_lock, no new dependency). The store lives in memory and is
rewritten wholesale at each checkpoint, so two handles on one store used to
silently destroy each other's data; a second writer now gets
MemoryError::Locked. The OS drops the lock with the descriptor, so a crash
never leaves a stale lock. Acquisition retries for ~250 ms to absorb a
previous owner that is mid-teardown; AsyncHDF5Memory::shutdown releases the
lock once its writer task has stopped.
- HDF5Memory::open_read_only: a lock-free, point-in-time view (checkpoint +
current WAL contents, replayed in memory) that never writes — it does not
repair, upgrade or move the WAL, and anything that would persist returns an
error. The CLI's recall/stats/agents-md/export use it, so a store can be
inspected while an agent has it open. Tests that reopened a store purely to
verify on-disk state now use it.
- open() no longer fails on a WAL that cannot possibly be replayed (torn
header, bad magic): it is moved to <store>.h5.wal.corrupt-<ts>, reported via
HDF5Memory::quarantined_wal(), and the healthy .h5 opens from its last
checkpoint. A well-formed header with an unknown version still fails and is
left untouched — most likely a newer build's WAL, which must not be
discarded.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Eight MemoryConfig fields (float16, compression, compression_level,
compact_threshold, hebbian_boost, decay_factor, wal_enabled, wal_max_entries)
were never written to /meta, so reopening a store silently reset them to
defaults — a compressed store was rewritten uncompressed by the first
checkpoint after a reopen, and wal_enabled=false flipped back to true. They
are now stored as /meta attributes; each is optional on load so older files
keep opening with the previous defaults, and non-finite floats are ignored.
Writing the round-trip test exposed that `compression = true` never worked in
a default build: the embeddings dataset called with_zstd() unconditionally but
the agent crate never enabled the zstd feature, so every checkpoint failed
with "unsupported filter: 32015". The default build now compresses with
deflate (always available, pure Rust path); Zstd is opt-in via a new `zstd`
agent feature.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
A save_or_update that hit an existing record was logged as a plain Save, so
replaying the WAL appended a duplicate instead of updating in place. It is now
logged as WalEntryType::Update (0x04) carrying the target index, and replay
applies it with cache.update().
The WAL header version goes 3 -> 4 for the benefit of older binaries: they
don't know record type 0x04, would read it as a torn tail and truncate it and
everything after it. An unknown header version makes them refuse the file
instead. The framing is otherwise identical, so v3 files are read by the same
code and upgraded in place on open (the header is outside the CRC chain).
Also drop the redundant WAL truncate that several callers ran straight after
flush(), which already truncates.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
flush() writes the new .h5 and only then truncates the WAL. A crash in that
window left a .h5 that already contained the pending entries AND a WAL that
still listed them, and open() replayed the WAL unconditionally — every pending
entry came back twice.
A checkpoint now records a WalMark in /meta (wal_applied_len/wal_applied_crc):
the byte length and chained CRC of the WAL prefix it folded in. On open, if
the WAL's v3 CRC chain passes through exactly that position, the entries up to
it are skipped; otherwise (the normal case: the WAL was truncated) everything
is replayed. No WAL format change; files without the attributes behave as
before. WalFile tracks its chain length alongside running_crc and resumes both
on reopen.
Also make the checkpoint and snapshot durable as a unit: sync the temp file
before the rename and the parent directory after it, so a power loss can't
leave an empty or partial .h5 under the final name. This is per-checkpoint
cost only; individual WAL appends remain unsynced by design.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
The norms guard was the tautology `n.len() == n.len()`, so a norms dataset
of any length was trusted and corrupted every cosine score; other per-record
datasets were not length-checked at all, so a truncated file loaded and then
panicked on the first index. Mismatches are now MemoryError::Schema, stored
norms are used only when they match the record count, and embedding_dim == 0
with records present is rejected instead of panicking in chunks(0).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Formatting only. cargo fmt --check was already failing on main (accel SIMD
kernels, agent, format, migrate, bench); CI now enforces it.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
- clippy --all-targets plus a clawhdf5-format feature matrix (parallel, lz4,
zstd, pcodec, fast-checksum); fix the accumulated lint backlog in test,
bench and feature-gated code (no behaviour changes).
- Install python3 + h5py/numpy/netCDF4/xarray in the CI container and set
CLAWHDF5_REQUIRE_INTEROP=1, which makes a missing interop dependency a test
failure. Every h5py/netCDF4 interop test used to skip silently in CI. Run
the #[ignore]d writer_h5py_tests suite explicitly.
- cargo bench --no-run so benches can't rot; fix bench.rs and memory_bench.rs,
which no longer compiled against the current strategy/consolidation APIs.
- Optional fuzz smoke run via CLAWHDF5_FUZZ_SECONDS.
- CHANGELOG and docs/known-issues.md updated.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Every test created its own wgpu instance and device (with adapter-maximum
limits) concurrently, which could wedge the driver and hang the suite
indefinitely. Tests now hold a process-wide lock while they own a device, and
GpuAccelerator readback waits are bounded at 30s so a stuck driver surfaces as
GpuError::BufferMap instead of blocking forever.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Compound datasets written with default libver bounds (datatype message
version 1, i.e. plain h5py.File(path, 'w')) could not be read: the v1 member
layout has 28 bytes of legacy array fields after the byte offset
(dimensionality 1, reserved 3, permutation 4, reserved 4, four sizes 16) and
the parser skipped 24, so every following member was read 4 bytes off. v2 was
also wrong: it keeps the 8-byte name padding and has no array fields.
Found by adding a default-libver axis to the h5py-generated-file tests (HDF5
2.0 raised the default low bound to 1.8, so "default" files are a distinct
format path from libver='latest'). Adds byte-level v1/v2 regression tests, a
truncation test, and fuzz corpus seeds for v1 compound and native complex.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Bump all workspace crates, the node package and pyproject to 2.2.0 and
finalize the changelog. The repository URL in every manifest pointed at a
GitHub location that does not resolve; use the real origin.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Class 11 (datatype version 5) properties are a single base floating-point
datatype message, not a compound-style member list. The old parser read the
base type's bytes as member names, yielding a garbage datatype, and failed
with UnexpectedEof when a complex type was nested in a compound.
Parse the base type and surface the type as the equivalent {r, i} compound
(the shape h5py writes for numpy complex dtypes), with a size check against
the base type. Covered by byte-level tests taken from HDF5 2.0 output and an
h5py end-to-end test (writer_h5py_tests is now 27/27 against HDF5 2.0.0).
Found while validating a user report of InvalidDatatypeVersion
{ class: 6, version: 5 } against v2.1.0 (already fixed on main in a13ff51,
never released). Add docs/known-issues.md recording that report, this bug,
the open reference-v4 gap and a gpu_tests parallel-run hang; credit the
reporter in the changelog.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
ann/io/migrate/agent work from three agent missions, an independent audit, and the two defects it found: the cosine near-zero guard weakened during the SIMD migration, and WAL appends after a torn tail being silently unreplayable. 52 suites green; both fixes proven by negative control.
The SIMD migration weakened the near-zero-norm guard in all four
clawhdf5-accel cosine_similarity backends (scalar/avx2/avx512/neon)
from `denom < f32::EPSILON` to `denom == 0.0`. Vectors with a tiny
but nonzero norm (denom in (0, 1.19e-7)) fell through to dot/denom
and scored as identical instead of maximally dissimilar, diverging
from the pre-SIMD scalar loop's documented fallback behavior.
Restores the epsilon threshold in all four backends so
`1.0 - cosine_similarity(...)` in hnsw.rs::compute_distance
reproduces the old fallback exactly. Adds regression tests in
clawhdf5-accel and clawhdf5-ann locking in the near-zero-norm case.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Two missions found largely disjoint ground — v3 in clawhdf5-ann, -io and
-migrate, v6 in -agent and -format — so this merge is mostly additive.
One genuine collision: BOTH runs independently implemented
`Dataset::verify_provenance` in clawhdf5/src/reader.rs, and git kept both,
producing `E0592 duplicate definitions`. They were functionally identical
apart from `self.file.data.as_bytes()` (v3) vs `self.file.as_bytes()` (v6).
Kept v6's — it is the variant that compiles against the current tree and
passed 52 suites, and its doc comment is the more honest one, stating both the
full-read cost and that an unkeyed hash stored beside its data is not a
tamper-evidence guarantee.
Verified present after the merge:
P1 clawhdf5-ann now depends on clawhdf5-accel; compute_distance calls
l2_distance / cosine_similarity instead of a scalar loop
P2 AsyncFileReader caches its handle and length behind a mutex
PR1 clawhdf5-migrate writes SHINES provenance, and validate checks it
v6 WAL torn-tail truncation, char-boundary truncate, agent hardening
52 suites pass, 0 failures.
Co-Authored-By: Claude Opus 5 <[email protected]>
Mission: 01a00c41-bac0-7eb3-a8c8-8b7044f3086d
Phase: 01a00c41-bac2-71e3-a58b-c473421200ee
Committed by the ClawMates delivery pipeline from the agents' working tree. Authored by agents, not by the named committer.
2026-08-16 20:44:28 +00:00
419 changed files with 102067 additions and 9266 deletions
Pure-Rust HDF5 format implementation with HNSW vector search, WAL-backed persistence, agent memory storage, and GPU-accelerated I/O. Used by ZeroClaw as its persistent memory and knowledge graph backend.
Pure-Rust HDF5 format implementation with HNSW vector search, WAL-backed persistence, agent memory storage, and GPU-accelerated vector search. A standalone library. Its one verified consumer is ClawBrainHub (`.brain` files); no agent framework integrates it (OpenClaw and ZeroClaw claims were withdrawn on 2026-09-25 — neither was ever true).
## Architecture
Cargo workspace with 16 crates under `crates/` (plus `libaec-sys`, an internal FFI bindings crate for the optional `szip` feature):
Cargo workspace with 19 crates under `crates/` (plus `libaec-sys`, an internal FFI bindings crate for the optional `szip` feature):
| Crate | Role |
|-------|------|
| `clawhdf5-format` | HDF5 binary spec parser (superblock, B-tree, heap) — also holds shared type definitions and physical constants |
| `clawhdf5-filters` | Deflate backends (zlib-rs, zlib-ng, Apple Compression); the HDF5 filter pipeline, the filter registry (`clawhdf5_format::filter_registry`) and the other codecs (LZ4, Zstd, SZIP, N-Bit, scale-offset, pcodec, and the pure-Rust plugin filters LZF, bitshuffle, bzip2, Blosc 1, and Blosc2 and ZFP read-only) live in `clawhdf5-format`. |
| `clawhdf5-derive` | Proc-macro derive for HDF5-serializable structs |
| `clawhdf5-wasm` | WebAssembly (wasm-bindgen) reader for the browser; demo in `examples/wasm-viewer/` |
| `clawhdf5-remote` | Remote files: `open_url` over HTTP(S) range requests and object stores (`object_store`: S3, GCS, Azure) through a mandatory block cache (`BlockCache`) |
| `clawhdf5-bench` | Benchmark suite |
## Key Features
- Zero-dependency HDF5 read/write (no libhdf5 C library required)
- Zero-C-dependency HDF5 read/write: no libhdf5, and deflate defaults to
pure-Rust zlib-rs (`fast-deflate` opts into zlib-ng, which needs cmake).
`ci-test.sh` fails if a C-building crate enters the core crates' default
tree. flate2 must keep `runtime_detection` with zlib-rs — without it zlib-rs
loses SIMD and inflates 3.5x slower. MSRV is 1.92 (`rust-version`, checked
in CI).
- HNSW vector index for semantic similarity search over agent memories — the
`clawhdf5-agent``hnsw` feature is **on by default**, so `hybrid_search` uses
the approximate `clawhdf5-ann` index for the vector stage (the index mirrors
the cache and self-heals on drift). Build the agent with
`--no-default-features --features float16` to force the exact linear cosine scan.
The agent's `parallel` feature (also default) builds the index on a thread
pool; the graph is identical with or without it.
The index uses the HNSW paper's diversity heuristic for neighbour selection
(plain closest-M capped recall on clustered data: 0.31 recall@10 at 100K). Its
graph is saved to `<store>.h5.ann` at each checkpoint and reloaded by `open()`
(tied to the checkpoint by a generation id; stale/damaged sidecars are
ignored and the index rebuilt). `MemoryConfig::quantized_index` (**on by
default** for new stores, persisted; stores predating the setting load as
`false` and keep their f32 index — guarded by
`tests/fixtures/store_v2_5_0.h5`; CLI opt-out is `create --f32-index`)
stores the index's own copy of the embeddings as `i8`,
which roughly halves a loaded store's memory (2.72x -> 1.74x the raw vectors
at 100K); because quantised distances are approximate and `ef` cannot
compensate, the query path then re-scores the candidate pool against the
exact embeddings, which holds recall at the f32 index's level. It is also
faster at equal recall: 1.63x the QPS on x86-64 (AVX2) and 1.18x on a
Raspberry Pi 5 (`clawhdf5_accel::dot_i8`, NEON `SDOT` via inline asm since
the intrinsic is unstable; plain NEON on pre-dotprod cores). The aarch64
code is `cfg`'d out on x86, so x86 CI never compiles or lints it — test it
on real ARM (`rpivision02`, 10.0.2.3, is a Pi 5). `hybrid_search` keeps one incremental BM25
index for the life of the store and never writes the store: Hebbian
activation boosts are persisted by the next checkpoint (or on drop), not per
query. Measure any search-path change with
`cargo run --release -p clawhdf5-bench --bin search_harness` (baselines in
`BENCHMARKS.md`).
- WAL (write-ahead log) for crash-safe persistence, with a chained CRC32
trailer per entry (each entry's CRC folds in the previous entry's CRC) so a
corrupted, reordered, duplicated, or spliced entry stops replay cleanly
@@ -40,6 +73,68 @@ Cargo workspace with 16 crates under `crates/` (plus `libaec-sys`, an internal F
format (v2) is still fully readable; the oldest no-CRC format (v1) is only
reachable through the one-time migration path in `HDF5Memory::open`, not
through the public `WalFile::read_entries`.
**What the WAL guarantees:** integrity, ordering, and recovery from a
*process* crash at any point — including between a checkpoint and the WAL
truncate (each checkpoint records a `WalMark` in `/meta`, and `open()` skips
the WAL prefix the `.h5` already contains, so entries are never applied
twice). Checkpoints and snapshots are made durable as a unit (temp file
synced, renamed, directory synced). **What it does not guarantee:**
individual WAL appends are *not* fsynced (a deliberate latency trade-off), so
saves made since the last checkpoint can be lost on power failure or kernel
panic. Current header version is 4 (adds the `Update` record used by
`save_or_update`); v3 files are read and upgraded in place.
- A store has a **single writer**: `HDF5Memory::create`/`open` hold an exclusive
advisory lock on `<store>.h5.lock` and a second opener gets
`MemoryError::Locked`. Use `HDF5Memory::open_read_only` for a lock-free,
never-writing point-in-time view (the CLI's `recall`/`stats`/`agents-md`/
`export` do). An unreadable WAL (torn header, bad magic) is quarantined to
`<store>.h5.wal.corrupt-<ts>` rather than blocking `open()`; a WAL with an
unknown *newer* version still fails and is left untouched.
-`MemoryConfig::float16` (**on by default** for new stores, persisted;
existing stores keep their recorded `false` — guarded by the v2.5.0
fixture in `tests/float16_store.rs`; CLI opt-out is `create --f32`) writes
`/memory/embeddings` as IEEE half precision (48% smaller file at 100K;
LongMemEval with real MiniLM embeddings identical to f32).
`MemoryCache::half_precision` rounds each embedding as it enters the cache (push, update, WAL replay, and on load of a store still
`f32` on disk), so memory and file agree bit for bit; the conversions live
in `clawhdf5_format::float16` and must stay the single implementation.
Values beyond ±65504 are `MemoryError::InvalidEntry`. Interop: every file
must open in h5py — `f32` datasets and empty datasets did not until
2026-09-23 (see `docs/known-issues.md`); the agent's `h5py_interop` test
guards a whole store.
-`HDF5Memory::search(query_emb, text, &SearchOptions)` is the full search
path: optional source-channel filter (applied before ranking; exact scan of
the allowed records whenever cheaper than `pool × M` index distance
evaluations, and as the fallback when the pool comes back short), fusion,
activation scaling, optional re-ranking and confidence rejection.
`hybrid_search`/`hybrid_search_with` are thin wrappers; `ClawhdfBackend`
(the `openclaw` module) is `search` with re-rank + confidence on.
- **OpenClaw is not supported** (decided 2026-09-25): clawhdf5 is not an
OpenClaw memory plugin and never was — the old `memory.backend = "clawhdf5"`
config was never valid. Don't reintroduce OpenClaw claims; `docs/openclaw.md`
records what a real plugin would need.
- **ZeroClaw does not use clawhdf5** (checked 2026-09-25 against upstream
v0.8.5 and the `osobh/zeroclaw` fork, and their full history): no
`clawhdf5` feature or backend exists; ZeroClaw's memory backends are
sqlite/lucid/postgres/qdrant/markdown/none behind its own `Memory` trait.
| limits | 20 s timeout (SIGKILL), 4096 MiB address space, per process; 16 files in parallel |
| runtime | 21 s probing + comparing (0 s fetch/build before it) |
## Results
A file's class is the first that applies:
- **panic / hang / crash / oom** — clawhdf5 panicked (caught per object or not), hit the timeout, died on a signal, or failed an allocation. The CI gate fails on any of these.
- **h5py-cannot-read** — libhdf5 could not open the file (or itself crashed or hung). Nothing to compare against; most are the deliberately malformed CVE reproducers.
- **our-error** — clawhdf5 returned an error for something h5py reads.
- **mismatch** — both read it, but the shapes, values, object set or attribute set differ.
- **ok** — every object h5py reads, clawhdf5 reads identically.
| corpus | files | ok | our-error | mismatch | h5py-cannot-read | panic | hang | crash | oom |
reads back in h5py as `[4.6e-41, 9.0e-44]`. Affected here: `NCAS-CMS_pyfive/tests/data/attr_datatypes.hdf5`, `hdf5/tools/test/testfiles/tcomplex_be.h5`.
- **Non-IEEE floats and partial-precision integers (N-Bit).** libhdf5 converts a float whose
bit layout is not IEEE (e.g. `H5Tset_precision` for the N-Bit filter) or an integer with a
bit offset / reduced precision into the plain numpy type of the same size. The probe
compares such values as converted numbers, not raw file bytes (before 2026-09-25 it compared
raw bytes, which reported every N-Bit float dataset as a mismatch).
- **Types h5py widens.** Where h5py reads a type into a numpy type of a different size
(FP8 -> float16, bfloat16 -> float32, x87 long double -> float128) the values are not
compared (shape and presence still are): dataset file type size 1 -> numpy float16 (2) (15x), attr file type size 1 -> numpy float16 (2) (15x), dataset file type size 2 -> numpy float32 (4) (2x), dataset file type size 8 -> numpy float128 (16) (1x), dataset file type size 12 -> numpy float128 (16) (1x), attr file type size 2 -> numpy float32 (4) (1x), dataset file type size 2 -> numpy >f4 (4) (1x), attr file type size 2 -> numpy >f4 (4) (1x).
- **Corrupt data HDF5 2.0 reads through a bug.** clawhdf5 refuses these objects; h5py 3.16 /
HDF5 2.0 returns values for them that the file does not hold:
-`cve_hdf5/cvefiles/cve-2025-2308.h5``/Scale_offset_long_long_data_le`: scale-offset codes run past the end of the chunk: HDF5 2.0 reads past its buffer; libhdf5's develop branch refuses the chunk ("Buffer too short").
-`cve_hdf5/cvefiles/cve-2025-44904.h5``/Scale_offset_float_data_le`: unfiltered chunks of 38 and 37 bytes for 48-byte chunks: HDF5 2.0 fills the rest with whatever its buffer held; libhdf5's develop branch refuses them ("incorrect chunk size returned from index for unfiltered chunk").
-`hdf5/test/testfiles/bad_nbit_parms_walk.h5``/Nbit_int_data_le`: an N-Bit parameter list one value short: HDF5 2.0 reads past the list; libhdf5's own test (`test_filter_bad_params`, test/dsets.c) now requires the read to fail.
- **References** are compared by presence only (`R`), not by target.
## Objects h5py fails on but clawhdf5 reads
- 19 x `OSError: Can't synchronously read data (no appropriate function for conversion path)`
- 1 x `TypeError: unhandled dtype kind M (dtype('…'))`
- 1 x `TypeError: No NumPy equivalent for TypeTimeID exists`
- 1 x `ValueError: Insufficient precision in available types to represent (N, N, N, N, N)`
- []**7.8**~~Documentation + migration guide~~ — removed: they described an integration that never worked
**Node.js bridge:**`clawhdf5-napi` (napi-rs) → `@redclaw/clawhdf5` npm package with full TypeScript types.
**Node.js bridge:**`clawhdf5-napi` (napi-rs) and a TypeScript wrapper in `packages/clawhdf5-node` exist but are unpublished, untested in CI and known to be broken (docs/known-issues.md).
---
> **Withdrawn.** None of this track produced a working OpenClaw integration: no
> plugin was built, the documented `memory.backend = "clawhdf5"` config was never
> valid in any OpenClaw release, and the Node package was never published. The
> Rust `ClawhdfBackend` remains as a library API. Not pursued for now; see
> [docs/openclaw.md](docs/openclaw.md) for what a plugin would need today.
| `fetch-corpus.sh` | shallow, sparse, blob-filtered checkout of each pinned commit into `.cache/src/` (gitignored); no-op when already there |
| `list_files.py` | which files are probed (HDF5/netCDF-4 extensions minus netCDF classic, plus the CVE reproducers) |
| `probe/` | the clawhdf5 side: a standalone crate (outside the workspace, so `cargo test --workspace` never builds it) that walks a file with `clawhdf5-format` and prints canonical JSON |
| `ref.py` | the h5py side: the same JSON from h5py |
| `run_one.sh` | runs both sides on one file (and `h5dump` on the CVE corpus) under a timeout and an address-space limit |
| `compare.py` | classifies each file (ok / our-error / mismatch / h5py-cannot-read / panic / hang / crash / oom) and groups root causes |
| `report.py` | writes `CONFORMANCE.md` |
| `check.py` | the gate: fails on any panic/hang/crash/oom, on an ok count below `baseline.json`, or on a baseline-ok file that is no longer ok |
| `baseline.json` | the ok files the gate holds the line on |
"comment":"conformance/run.sh fails if the ok count drops below `ok` or a file in `ok_files` stops being ok. Regenerate with `conformance/run.sh --update-baseline` after an intended change.",
w(f"| limits | {meta_run.get('timeout_s')} s timeout (SIGKILL), {int(meta_run.get('mem_kb',0))//1024} MiB address space, per process; {meta_run.get('jobs')} files in parallel |")
w(f"| runtime | {meta_run.get('probe_seconds')} s probing + comparing ({meta_run.get('build_seconds')} s fetch/build before it) |")
w("")
w("## Results")
w("")
w("A file's class is the first that applies:")
w("")
w("- **panic / hang / crash / oom** — clawhdf5 panicked (caught per object or not), hit the timeout, died on a signal, or failed an allocation. The CI gate fails on any of these.")
w("- **h5py-cannot-read** — libhdf5 could not open the file (or itself crashed or hung). Nothing to compare against; most are the deliberately malformed CVE reproducers.")
w("- **our-error** — clawhdf5 returned an error for something h5py reads.")
w("- **mismatch** — both read it, but the shapes, values, object set or attribute set differ.")
w("- **ok** — every object h5py reads, clawhdf5 reads identically.")
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
osobh