Every structure in both chunk indexes — header, index block, super
block, data block and each data block page — carries a Jenkins lookup3
checksum, and all of them were parsed past and ignored.
What that costs is not a warning but correct data. Flip one low bit of a
chunk address and the index still has the right shape, the address still
lands inside the file, and the reader returns whatever bytes now sit
there as that chunk's contents. Nothing else in the parse can tell.
Verified in both directions. The checksums accept files written by
HDF5 2.0 from 100 to 200 000 chunks — dense, sparse, gzip-filtered and
paged — which also confirms the block layouts byte for byte, since a
wrong offset would fail every file. And an interop test corrupts an
address to check the read fails instead of returning data: removing the
verification makes that test fail with "corruption produced data instead
of an error", which is what it is there to prove.
The first version of that test passed with verification disabled — it
corrupted a byte a structural check already rejected, so it proved
nothing. Worth recording, since a test that passes for the wrong reason
looks exactly like coverage.
Hand-built fixtures now stamp real checksums, as HDF5 writers do, and
the Extensible Array ones no longer describe the superseded layout.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
A dataset with exactly one unlimited dimension — the ordinary
append-only case — is indexed by an Extensible Array. Only its first
few chunk entries (4 by default) sit inline in the index block, and
everything past them was read with the wrong layout. In the default
shape the 37th chunk onward came back from the wrong place: a
400-chunk dataset returned 364 wrong values while reporting success,
and beyond about a thousand chunks the read failed outright. Silently
wrong data is the worse half of that.
It survived because the only Extensible Array fixture in the suite had
three chunks — inside the inline limit — so no test ever reached a data
block.
Four layout errors, each confirmed against files written by HDF5 2.0 and
against the library source rather than inferred:
- super block `u` owns 2^(u/2) data blocks, not 2^u;
- each holds 2^((u+1)/2) * data_blk_min_elmts elements — the two
quantities double every *other* level, a half step apart;
- a super block carries a block-offset field before its data block
addresses, which was not skipped;
- the page-init bitmap belongs to the super block, one bit per page
packed across all of its data blocks and read MSB-first, rather than
living inside the data block; a paged data block also ends its prefix
with a checksum before the first page.
Where the spec left room for doubt the file settled it: decoding a
paged block's elements and reading the chunk values they address
identifies the mapping exactly, and the bitmap's 68 set bits matched
the 34 data blocks x 2 pages that 200 000 elements need, which only
holds MSB-first.
New interop tests cross every boundary — 4, 37, 400, 5 000 and 200 000
chunks, the last with paged data blocks — plus sparse (uninitialised
pages taking fill values), gzip-filtered elements and a 2-D dataset.
All three fail against the old traversal.
Writing is untouched; this was a read-path bug.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The existing target only called `BTreeV2Header::parse`, so the
recursive walk behind it — where a node that is its own child overflowed
the stack — was never fuzzed at all. Parsing also requires a valid
Jenkins checksum, which random input essentially never produces, so
almost every input stopped at the first branch.
The target now walks the tree after a successful parse, and also builds
a header straight from the input bytes so the traversal is reachable
without forging a checksum.
Checked both ways: against the unfixed traversal libFuzzer finds the
stack overflow (ASan: stack-overflow), and against the fix that same
input executes in 0 ms and 34.7 million further runs produce no crash,
timeout or OOM.
Corpora and crash artifacts stay out of the repository; the two crafted
inputs are covered by unit tests instead.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Traversal recursed one frame per level with the depth taken from the
file (a u16), and followed child addresses without asking whether they
were shared. Two crafted inputs, both reproduced before fixing:
- A node listing itself as its own child, under a header claiming 65 535
levels, overflowed the stack and aborted the process — SIGABRT, not an
error a caller can handle — from under 100 bytes.
- Levels whose children all point at one shared node below reached it
fan-out^depth times: 29.5 million records in 8 s from ~5 KB, and one
more level would exhaust memory.
Depth is now capped at 64, as the fractal heap already was; no real tree
approaches it, since even at the minimum fan-out of two that is over
2^64 records. And traversal stops once it has produced more records
than the file has bytes to hold them — a valid tree stores each record
once in its own bytes, so this bounds shared subtrees without trusting
the header's own `total_records`. Both inputs now fail in under a
millisecond.
Every B-tree v2 user goes through this collector: dense attributes, v2
groups, shared messages and chunk indexes. To show the budget never
refuses a real file, a new interop test has HDF5 2.0 write a depth-2
chunk index with 40 000 records and reads back all 160 000 values; it
fails when the budget is deliberately made too tight.
Also corrects `BM25Index::search`, which claimed to use Block-Max WAND.
It scores exhaustively, and pruning would not help the store:
`hybrid_search` needs every score because fusion normalises over them.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Every Python interop suite had stopped running on this machine: the h5py
writer round-trips, the facade suite, netCDF4 and the reference files.
`python3` is 3.14, nothing on the box has h5py, and PEP 668 refuses to
install it into a system interpreter at all — so the availability probes
all returned false and each suite skipped without failing.
A silent skip here is exactly how the v5 compound-datatype bug reached a
release, so the probes now read `CLAWHDF5_PYTHON` and `ci-test.sh` picks
up `.venv/bin/python` on its own. The detection sits at the top of the
script rather than beside the interop step, because the non-ignored
suites run in the earlier `cargo test` step and would otherwise still
miss it. `CLAWHDF5_REQUIRE_INTEROP=1` continues to turn a skip into a
failure.
Verified against a venv with h5py 3.16 / HDF5 2.0.0: 94 interop tests
across the four suites, all passing.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Bump all workspace crates, the node package and pyproject to 2.5.0, fold the
two unreleased sections together and add upgrade notes for the behaviour
changes.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
HDF5 1.12 revised the reference datatype (class 7) in datatype message version
4: reference types 2-4 are the new H5T_STD_REF object / dataset-region /
attribute references. Datatype::parse rejected them with
InvalidReferenceType, so any dataset of that type was unreadable.
h5py cannot write this type, which is why it had never been tested. A real
file was produced by calling the libhdf5 bundled in the h5py wheel through
ctypes (H5T_STD_REF_g, H5Rcreate_object, H5Dwrite); the 2 KB result is
committed as tests/fixtures/std_ref_hdf5_2_0.h5 with its generator,
gen_std_ref.py.
- ReferenceType gains Object2, DatasetRegion2 and Attribute, accepted only
from datatype version 4.
- read_object_references decodes Object2 elements: type(1) flags(1)
token_size(1) token, zero-padded to the element size; the token is the
target's object header address. A null reference decodes to the undefined
address; an external reference, a wrong type byte or a token that doesn't
fit is an error.
The fixture test follows both references and checks they resolve to the
objects they were created from.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Requesting a filter without chunk dimensions made the whole dataset a single
chunk. Any read, even one row, then decompresses everything, and a large
dataset cannot be decoded in parallel — which also made the new partial reads
pointless for such files.
auto_chunk_dims keeps datasets up to 1 MiB as one chunk (unchanged behaviour)
and splits larger ones by halving the dimensions in turn, so chunks keep
roughly the dataset's proportions, until a chunk is at most 1 MiB — h5py's
approach. An empty (unlimited, unwritten) dimension is treated as 1024. The
writer passes the element size through resolve_chunk_dims_for; the old
resolve_chunk_dims assumes 8-byte elements. Explicit with_chunks always wins.
Interop test: h5py reads an auto-chunked 13 MB deflate dataset, sees chunks
between 128 KiB and 1 MiB, and a small dataset still has one chunk.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
shuffle_decompress — on the read path of every compressed dataset, since
shuffle is applied automatically before compression — was the naive
`result[i * es + j] = data[j * n + i]`: a multiply and two bounds checks per
byte. It now interleaves fixed-width arrays of byte planes for element sizes
2/4/8/16 (bounds checks hoisted, vectorisable), with a chunked generic
fallback. The write-side shuffle was already optimised; this was the asymmetry
the survey flagged. Modest wall-clock effect now that decode is parallel
(chunked+deflate full read ~70 -> ~66 ms). Round-trip test over element sizes
1-24 and several lengths.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Same-moment A/B on a 64 MB f64 dataset: chunked+deflate 110 -> 69 ms, chunked
72 -> 60 ms, contiguous 56 -> 30 ms.
- read_chunked_data_cached — the path the facade uses — decompressed chunks
one at a time; only the uncached reader was parallel. Cache misses are now
decoded in bounded batches (128), in parallel with the `parallel` feature.
- Every chunk was pushed into the 16 MiB chunk cache, which a larger dataset
just churns (insert, evict moments later). Chunks are cached only when the
whole dataset fits (new ChunkCache::max_bytes).
- Unfiltered chunks went file -> Vec -> aligned cache buffer -> output. They
are copied straight from the file bytes.
- The facade's typed reads convert a contiguous dataset straight from the
borrowed file bytes instead of copying it into a Vec first.
- The native little-endian fast paths allocated vec![0; n] and then overwrote
it; they now fill an uninitialised buffer in one copy (native_le_to_vec).
alloc_output requests zeroed memory from the allocator instead of reserving
and filling.
The unit test that expected unfiltered chunks to land in the decompressed
cache now asserts the new design (index reused, cache not involved).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
With libver='latest', a chunked dataset with two or more unlimited dimensions
indexes its chunks with a v2 B-tree (layout v4, index type 5). Reading one
failed with "unsupported chunked layout version=4, index_type=Some(5)".
read_btree_v2_chunks decodes record types 10 (address + scaled offsets) and 11
(address, stored size, filter mask, scaled offsets). The width of the
stored-size field is taken from the record size the tree header declares
rather than re-deriving the library's formula. Scaled offsets are multiplied
back by the chunk dimensions with overflow checks.
The chunk-index dispatch existed four times (uncached, cached, sweep and
indexed readers). The three copies outside list_chunks now call it, so every
read path — and fill-value handling and partial reads — supports every index
type from one place.
h5py interop test: plain, gzip+shuffle, a 2500-chunk tree with internal nodes,
a sparse dataset with a fill value, and a strided hyperslab.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
read_raw_data_selection computed which chunks a selection intersects, threw
the answer away, decoded the entire dataset and picked elements out of it —
for contiguous layouts too. A 64x64 window of a 64 MB deflate dataset cost
105 ms, about half a full read; every selection cost the same whatever its
size.
New partial_read module: materialise only the selection's bounding box — the
overlapping rows of a contiguous dataset (straight from the file bytes) or the
overlapping chunks (only those are decompressed) — then run the existing
extractor over that buffer with the selection translated to the box origin, so
extraction semantics are exactly the full-read ones. It declines (falling back
to the old path) for All/None, compact/virtual/storage-less layouts, and boxes
covering more than half the dataset. That window now takes 0.39 ms, one row
2.7 ms, one column 5.2 ms.
Selections are validated against the dataset shape first. They were not: a
hyperslab past an edge came back padded with zeros and a point with an
out-of-range column wrapped into the next row, returning the wrong element
with no error. Now FormatError::SelectionOutOfBounds (also rank mismatch and
overlapping blocks); the facade's fill-aware path validates too.
Tests: equivalence against a reference extraction from a full read over 60
random hyperslabs/point lists per layout (contiguous, chunked, deflate) for
ranks 1-3. New read_harness bench binary with before/after in BENCHMARKS.md.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Bump all workspace crates, the node package and pyproject to 2.4.0, finalize
the changelog and add upgrade notes for the search behaviour changes.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Bump all workspace crates, the node package and pyproject to 2.3.0, finalize
the changelog and add upgrade notes for the behaviour changes.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
attrs() silently omitted any attribute whose datatype had no AttrValue variant
— including every Python bool, which h5py stores as an enum — plus complex,
compound and reference attributes, and cast unsigned 64-bit arrays to
I64Array so values above i64::MAX came back negative.
- numpy/h5py-style booleans (an enum of exactly FALSE=0 / TRUE=1 over an
integer base) decode as I64 / I64Array of 0/1.
- AttrValue::U64Array keeps unsigned arrays unsigned. Behaviour change: an
unsigned array attribute no longer arrives as I64Array; the netCDF-4 CF
helpers (_FillValue, valid_range) and the Python bindings handle it.
- AttrValue::Raw { datatype, shape, data } carries any other attribute
verbatim (also used when a value fails to decode as its declared type), so
the attribute list is always complete. Decodable with data_read against the
datatype; Python receives {"dtype", "shape", "data"}.
- Both new variants are writable, so attributes round-trip between files.
h5py interop tests cover reading 13 attribute kinds and h5py reading back a
compound and a u64 attribute written by clawhdf5.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
- Path resolution follows soft links in both old-style (symbol table, cache
type 2) and new-style (compact and dense Link message) groups: absolute and
relative targets, links to groups, links through links, with a depth limit
so a link cycle is NestingDepthExceeded rather than a hang. A dangling link
reports the target it could not find. Previously every soft link was
PathNotFound.
- An external link is FormatError::ExternalLinkUnsupported { filename,
object_path } instead of a misleading PathNotFound.
- Message 0x0007 (External Data Files) is now a known MessageType, and a
dataset carrying it is FormatError::ExternalDataFilesUnsupported. Such a
dataset has no data address in this file, so it would otherwise be read as
"never written" and answered with fill values — wrong data, no error.
- Dense link iteration is shared between hard-link listing and the new
symbolic-link lookup; entry listing behaviour is unchanged.
- h5py interop test for both libver settings.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
HDF5 allocates lazily: a chunk nobody wrote doesn't exist in the file, and a
dataset nobody wrote has no data address. Such regions must read as the
dataset's fill value. There was no Fill Value message parser at all, so:
- a sparse chunked dataset read its holes as zeros — silently wrong whenever
the fill value isn't zero (h5py `fillvalue=-1` came back as 0);
- a dataset that was created but never written failed with NoDataAllocated /
"no address for chunked layout" where h5py returns a filled array.
New clawhdf5_format::fill_value: parses Fill Value messages v1-v3 and the old
0x0004 message (validated against HDF5 2.0 output under default and latest
libver), builds a fully filled dataset when there is no storage, and writes the
fill value into exactly the chunk-grid cells absent from the chunk index —
never mistaking a stored zero for a hole, clipping edge chunks, any rank. It is
skipped entirely for the default (zero) fill value. The chunk index dispatch is
extracted from read_chunked_data into a reusable list_chunks.
The reader, lazy and mmap facades apply it on full reads; selection reads go
through a fill-aware full read when the fill value matters. h5py interop test
compares against h5py's own readback, including a sparse 2-D dataset and a
hyperslab straddling allocated and unallocated chunks.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
A dataset created from a committed (named) datatype stores only a shared-
message reference to it. The facade parsed those reference bytes as the
datatype itself, producing `Time { size: 0 }` and unreadable data, and an
attribute using a committed datatype was silently dropped.
- shared_message::parse_shared_ref had the encoding wrong: it skipped six
reserved bytes for version 2 (only version 1 has them) and had the version 3
types inverted (1 is the SOHM heap, 2 is "committed, in another object
header"). Verified against h5py 3.16 / HDF5 2.0, which writes
`02 02 <address>` under both default and latest libver bounds. Resolution
now dispatches on which field the reference carries.
- New shared_message::message_data resolves a header message through the
indirection; the reader, lazy and mmap facades use it for datatype,
dataspace and filter-pipeline messages.
- AttributeMessage honours the v2/v3 flags (bit 0 datatype shared, bit 1
dataspace shared) via the new parse_in_file, used everywhere file data is
available. Parsing a shared attribute without file access is now
FormatError::UnresolvedSharedMessage instead of a garbage datatype.
- h5py interop test covering both libver settings.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Dataspace and chunk dimensions are untrusted 64-bit fields, but the chunked
read paths computed `num_elements() as usize * elem_size` and
`chunk_dims.product() * elem_size` with plain arithmetic and fed the result to
`vec![0u8; n]`. A crafted file could wrap the product (under-sizing the output
buffer that chunks are then copied into) or request an allocation large enough
to abort the process.
- Dataspace::checked_num_elements, checked_byte_len, checked_chunk_byte_len
and alloc_output (try_reserve_exact) replace the plain products and
vec![0; n] at every chunked read site, plus the VDS and hyperslab paths.
Overflow and allocation failure are FormatError::Overflow.
- Dataspace::num_elements saturates instead of wrapping.
- A zero-element dataset returns early, which also keeps the stride products
in range when another dimension is huge.
- parallel_read.rs: the three `c_addr + size > len` bounds checks used a raw
add; they now use checked_add like the rest of the crate.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Formatting only. cargo fmt --check was already failing on main (accel SIMD
kernels, agent, format, migrate, bench); CI now enforces it.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
- clippy --all-targets plus a clawhdf5-format feature matrix (parallel, lz4,
zstd, pcodec, fast-checksum); fix the accumulated lint backlog in test,
bench and feature-gated code (no behaviour changes).
- Install python3 + h5py/numpy/netCDF4/xarray in the CI container and set
CLAWHDF5_REQUIRE_INTEROP=1, which makes a missing interop dependency a test
failure. Every h5py/netCDF4 interop test used to skip silently in CI. Run
the #[ignore]d writer_h5py_tests suite explicitly.
- cargo bench --no-run so benches can't rot; fix bench.rs and memory_bench.rs,
which no longer compiled against the current strategy/consolidation APIs.
- Optional fuzz smoke run via CLAWHDF5_FUZZ_SECONDS.
- CHANGELOG and docs/known-issues.md updated.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Compound datasets written with default libver bounds (datatype message
version 1, i.e. plain h5py.File(path, 'w')) could not be read: the v1 member
layout has 28 bytes of legacy array fields after the byte offset
(dimensionality 1, reserved 3, permutation 4, reserved 4, four sizes 16) and
the parser skipped 24, so every following member was read 4 bytes off. v2 was
also wrong: it keeps the 8-byte name padding and has no array fields.
Found by adding a default-libver axis to the h5py-generated-file tests (HDF5
2.0 raised the default low bound to 1.8, so "default" files are a distinct
format path from libver='latest'). Adds byte-level v1/v2 regression tests, a
truncation test, and fuzz corpus seeds for v1 compound and native complex.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Bump all workspace crates, the node package and pyproject to 2.2.0 and
finalize the changelog. The repository URL in every manifest pointed at a
GitHub location that does not resolve; use the real origin.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Class 11 (datatype version 5) properties are a single base floating-point
datatype message, not a compound-style member list. The old parser read the
base type's bytes as member names, yielding a garbage datatype, and failed
with UnexpectedEof when a complex type was nested in a compound.
Parse the base type and surface the type as the equivalent {r, i} compound
(the shape h5py writes for numpy complex dtypes), with a size check against
the base type. Covered by byte-level tests taken from HDF5 2.0 output and an
h5py end-to-end test (writer_h5py_tests is now 27/27 against HDF5 2.0.0).
Found while validating a user report of InvalidDatatypeVersion
{ class: 6, version: 5 } against v2.1.0 (already fixed on main in a13ff51,
never released). Add docs/known-issues.md recording that report, this bug,
the open reference-v4 gap and a gpu_tests parallel-run hang; credit the
reporter in the changelog.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Datatype::parse recurses into itself for Compound/Enumeration/
VariableLength/Array/Complex member and base types with no depth
counter. A message data size capped at u16::MAX (65535 bytes) allows
~8000 levels of nesting in a crafted file, enough to blow the stack —
worse on the project's no_std/embedded targets with only a few KB of
stack. Thread a depth counter through a new parse_with_depth, mirroring
object_header.rs's continuation-depth guard, and reject past 64 levels
with FormatError::NestingDepthExceeded. The public Datatype::parse
signature is unchanged.
INT-03
SymbolTableNode::parse used raw offset+8 arithmetic that can overflow
on a crafted v1-group B-tree leaf with a near-u64::MAX SNOD child
pointer (group_v1.rs passes such offsets through unchecked). Switch to
checked_add, matching read_offset in the same file. Also harden the
entries_start + num_symbols*entry_size computation with checked_add
for consistency, even though num_symbols being u16 already bounds
that multiply. Add regression tests.
INT-02
Six sites used raw `offset + N > file_data.len()` arithmetic that can
overflow on a crafted file with an address field near u64::MAX,
bypassing the bounds check before the next slice op panics. Switch to
the checked_add-based ensure_len pattern already used by local_heap.rs
and other parsers in this crate. Add regression tests for offsets near
usize::MAX in both files.
INT-01
Found via a real-world integration audit against omni-cortex (a JEPA-based
cognitive architecture built on clawhdf5 as its tiered Working/Episodic/
Semantic memory store).
- Add DatasetBuilder::with_u64_data (crates/clawhdf5-format/type_builders.rs).
The read side already has read_u64/read_as_u64, but there was no
symmetric write-side builder — only signed with_i32_data/with_i64_data
existed. Every consumer needing full-range u64 (timestamps, IDs) had to
bit-cast through i64 via `i64::from_ne_bytes(v.to_ne_bytes())` on write
and reverse it on read. omni-cortex does this in at least 6 places
across its writer/reader/mmap-reader/consolidate crates. Confirmed the
new builder round-trips full-range u64 (including values with the high
bit set) end-to-end in a standalone sanity check mirroring their usage.
- Fix Dataset::read_selection(&Selection::All) to route through the same
per-file chunk cache read_raw()/read_f64() etc. already use, instead of
the uncached read_chunked_data path. Selection::All is semantically a
full read; there's no reason two ways of asking for "everything" should
have different caching behavior. Also gains read_raw()'s virtual-dataset
resolver support for free. omni-cortex's Reader/mmap-reader/consolidate
crates all call read_selection(&Selection::All) for their chunked/
compressed dataset reads, so this was a real, if currently low-traffic
(single-pass read pattern), inconsistency in the public API's behavior.
- README: fix a stale crate-map claim that clawhdf5-filters supports
"blosc" compression — it never did (the crate only ever held
fast_deflate.rs; lz4/zstd/pcodec/szip filters live in clawhdf5-format).
New tests: u64_data_roundtrip, read_selection_all_matches_read_raw_on_chunked_dataset.
Add [workspace.dependencies] to the root Cargo.toml for the four
duplicated-across-many-crates dependencies flagged by the earlier review:
tempfile (7 crates), criterion (6), half (4 — real version skew, clawhdf5-gpu
pinned 2.7 while others used bare 2), and serde (4). Update every consuming
crate to `dep = { workspace = true }`, preserving crate-local `optional =
true` where it already existed. half now resolves uniformly to 2.7.x
workspace-wide instead of two separate semver ranges.
Also fixed clawhdf5-filters/Cargo.toml's stale "rustyhdf5" description
while touching the file (same class of leftover rename as prior fixes).
Not touching rayon/byteorder/clap (no skew found, lower priority).
- Add ensure_len(data, offset, needed) helper to chunked_read.rs,
data_read.rs, and local_heap.rs (matching the existing btree_v1.rs/
object_header.rs convention) and use it at every plain-arithmetic
offset+size bounds check found in these files, closing usize-overflow
panics reachable from crafted near-usize::MAX offsets/addresses.
- collect_chunk_info: add a depth-limited internal wrapper
(collect_chunk_info_inner, MAX_CHUNK_BTREE_DEPTH=64) to reject a
crafted self-referencing/cyclic B-tree v1 chunk index instead of
recursing unboundedly (stack-overflow DoS).
- read_compound_fields: validate byte_offset+field_size against the
compound's declared element size before slicing, instead of an
unguarded out-of-bounds panic on a crafted member offset.
- read_chunked_data/_cached/_sweep/_indexed: guard `ndims - 1` against
underflow for a degenerate zero-dimension chunked layout.
- copy_chunk_to_output: rewrite all offset/stride arithmetic (both the
1-D fast path and the general N-D path) to use checked_add/checked_mul,
skipping an out-of-range row/chunk instead of panicking on overflow.
Add a new cargo-fuzz target, fuzz_dataset_read, that walks every dataset
in a parsed file via the clawhdf5 facade and exercises the contiguous/
chunked/compact raw-data read paths that the existing fuzz_full_file
target doesn't reach. Seeded with the chunked/VDS/compound-relevant test
fixtures plus two crash regressions found during this pass (the
copy_chunk_to_output overflow and the ndims-1 underflow, both fixed
above — this target found real bugs within the first couple of runs).
Not wired into CI (nightly-only, multi-minute runs); documented in
fuzz/README.md as a manual/scheduled check instead. Also fixed the
README's stale rustyhdf5-format naming while touching this file.
Added regression tests for every fix (near-usize::MAX offsets, the
self-referencing B-tree case, the compound byte_offset overrun, the
zero-dim layout, and both copy_chunk_to_output overflow paths) so these
are caught by `cargo test`, not just the fuzz corpus.
- Fix version skew: clawhdf5-py (pyproject.toml 1.93.0 -> 2.1.0) and
packages/clawhdf5-node (package.json 2.0.0 -> 2.1.0) were both behind
the actual crate version.
- Correct stale ROADMAP.md claims: the TypeScript bridge already has a
complete napi-rs package (not "no package.json"); CI/CD is now wired
up via .gitea/workflows/ci.yml.
- Fix CLAUDE.md: clawhdf5-gpu uses wgpu with hand-written WGSL compute
shaders, not CubeCL.
- chunked_read.rs: drop 12 unnecessary chunk_dimensions[..rank].to_vec()
allocations — all three callees already accept &[u32].
- btree_v1.rs: add an overflow-safe ensure_len(data, offset, needed)
helper (checked_add) and use it at the two plain-arithmetic bounds
guards, closing a usize-overflow edge case reachable from a crafted
near-usize::MAX B-tree offset. Add a regression test.
- Clarify that the integrity hashes in clawhdf5-agent/provenance.rs
(FNV-1a) and clawhdf5-format/provenance.rs (SHA-256) are unkeyed and
only detect accidental corruption, not tampering — doc-only change.
- README.md: document that the mpi-io feature's read/write paths are
root-read+broadcast / gather-to-rank-0, not true collective I/O.
- Add .gitea/workflows/ci.yml running scripts/ci-test.sh (fmt, clippy,
test, no_std check) on push/PR to main.
- Fix stale rustyhdf5-py/rustyhdf5-format package names in
ci-test.sh/check-nostd.sh, which had been silently no-op'ing those
checks (cargo warns but doesn't fail on an unknown --exclude/-p
target).
- With those checks actually running, fix the real issues they surface:
- clippy: useless_conversion in chunked_write.rs, byte_char_slices in
global_heap.rs/object_header.rs.
- cargo fmt: apply formatting across the workspace (whitespace only).
- no_std (thumbv7em-none-eabihf) build errors in clawhdf5-format:
core::sync::atomic::AtomicU64 doesn't exist on that target (no
native 64-bit atomics) — switch profiling.rs's counters to
portable-atomic, which falls back to a CAS-based emulation there
and is a no-op wrapper elsewhere. Add missing alloc imports for
Box (filters.rs), Vec (filters_szip.rs), and format! (dict_encoding.rs)
on no_std paths. Replace f64::powi (std/libm-only) with a small
local exponentiation-by-squaring helper in the scale-offset filter.
The decompressed-chunk LRU cache was the hottest path in the read pipeline
(every chunked-dataset read goes through it) but did a linear scan through
up to 521 slots on every get/put, and a full buffer copy on every cache hit
(to_vec()/clone() of the whole decompressed chunk). chunked_read.rs then
cloned the buffer a second time just to insert it into the cache after
already having it in hand.
- Added a HashMap<ChunkCoord, usize> index alongside the LRU slots for O(1)
lookup. Eviction uses swap_remove, so the swapped-in slot's index entry is
fixed up on every eviction (covered by a dedicated test).
- CachedChunk.data is now Arc<CacheAlignedBuffer> — a cache hit is a
refcount bump, not a copy. CacheAlignedBuffer gained a Sync impl (same
soundness argument as its existing Send impl: access is only ever through
borrow-checked &/&mut, like Vec<u8>) so Arc<CacheAlignedBuffer> is itself
Send/Sync.
- put_decompressed/put_decompressed_aligned now return the Arc they just
inserted (or the existing cached copy), so callers can reuse that
allocation instead of holding a separate clone — eliminates the second
copy in chunked_read.rs's three call sites, which now consume the
Arc<CacheAlignedBuffer> (Deref's to &[u8], so downstream indexing/copy
code is unchanged).
- prefetch_hint's doc comment now leads with "bookkeeping only, does not
prefetch" instead of describing behavior it doesn't have.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
decompress_chunk() already threaded chunk_size (the pipeline's declared
decompressed size) into the scale-offset/nbit/szip decoders to bound their
output, but not into deflate/lz4/zstd/pcodec, all four of which allocated
based on attacker-controlled input with no cap:
- lz4: read a raw u32 "orig_size" straight from the compressed payload's
first 4 bytes and passed it directly to lz4_flex::block::decompress with
no upper bound — a 4-byte attacker-controlled field could request ~4 GiB.
- deflate (non-macOS path): unbounded flate2 read_to_end into a fresh Vec.
- zstd: zstd::decode_all with no output cap (classic decompression-bomb
vector, ratios can exceed 1000:1).
- pcodec: simple_decompress with no cap.
All four now take the expected chunk size and reject output that exceeds it
(or a 256 MiB absolute ceiling when the size is unavailable), matching the
pattern the other three filters already used. Also fixes the same unbounded
read_to_end in clawhdf5-filters' fast_deflate streaming fallback (used when
no size hint is available).
Added tests for each codec plus one exercising the actually-exploited path
through the public decompress_chunk() entrypoint.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
- Remove clawhdf5-types (empty 1-line stub crate; type defs already live in
clawhdf5-format). Update workspace Cargo.toml and CLAUDE.md accordingly.
- Implement HDF5 superblock v4 (page-buffer mode) read and write support in
clawhdf5-format: Superblock::parse_v4, page_size field, v4 serialize
branch, and FileWriter::with_page_size. This was the one task left
unimplemented from docs/superpowers/plans/2026-06-29-format-write-extensions.md.
- Reconcile the three docs/superpowers/plans/*.md docs (filter codecs,
format write extensions, MPI-IO VOL) against actual shipped code: they
were pre-work plans for d6c4d4f (2026-06-30) committed to git late on
2026-08-03 with all checkboxes still unchecked. Mark completed tasks done
and add a status note so they read as historical records, not open work.
- Refresh ROADMAP.md's "What's Next" section against current repo state.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Enables Rayon parallel compression for typical 4-chunk workloads (e.g.,
128×128 matrix with 32-row chunks). Rayon's dispatch overhead is ~2 µs,
worthwhile at ≥3 chunks with real compression work per chunk.
Previously the threshold was "> 4" which excluded 4-chunk datasets entirely
from parallel compression. Now "> 2" covers 3+ chunks.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Following arXiv:2506.18062 (TDT pre-filter) and matching h5py default
behavior: the shuffle filter is now automatically applied before any
compression codec (deflate, Zstd, LZ4, Pcodec) unless explicitly
disabled with .without_shuffle().
Benchmark results (f32 matrices, shuffle+codec vs unshuffled baseline):
- Zstd-3 at 512×512: 610 → 764 MiB/s (+25%)
- Deflate-6 at 128×128: 132 → 401 MiB/s (+204%)
- Deflate-6 at 512×512: 280 → 745 MiB/s (+166%)
Both codecs now reach parity at ~750 MiB/s for large matrices.
Changes:
- Add no_shuffle field to ChunkOptions (opt-out via .without_shuffle())
- Auto-add FILTER_SHUFFLE in build_pipeline() when compression is active
- Add DatasetBuilder.without_shuffle() method
- Update pipeline tests to reflect new 2-filter default
- Add chunk_options_pipeline_deflate_no_shuffle test
- Update BENCHMARKS.md with measured throughput improvements
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Implements Pcodec (filter ID 32023) via the `pco` 1.0.x crate as a new
optional compression codec. Pcodec achieves 30–94% better compression
ratio than Zstd for f32/f64 columnar data at 1–5 GiB/s decompression
speed, making it ideal for write-once/read-many embedding archives.
Write throughput at 512×512: 591 MiB/s (parity with Zstd-3 at 610 MiB/s).
For smaller chunks Zstd-3 remains faster due to Pcodec's fixed per-chunk
distributional analysis overhead.
- Add FILTER_PCODEC = 32023 constant to filter_pipeline.rs
- Add pcodec_compress/pcodec_decompress using pco::standalone API
- Wire into compress_chunk/decompress_chunk dispatch
- Add ChunkOptions.pcodec field and DatasetBuilder.with_pcodec() method
- Enable pcodec as highest-priority codec in build_pipeline()
- Add pco dep (optional, feature = "pcodec") to clawhdf5-format/clawhdf5
- Add write_2d_chunked_pcodec benchmark comparing pcodec vs zstd-3
- Document results in BENCHMARKS.md
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Four independent write-path improvements:
1. Cache compressed chunks between Pass 1 and Pass 2 (chunked_write.rs,
file_writer.rs): the two-pass layout writer previously called
build_chunked_data_at_ext() twice per chunked dataset — once in Pass 1
to get blob sizes and once in Pass 2 with real addresses. Add
PrecompressedChunks / precompress_chunks() / build_chunked_data_from_
precompressed() to compress once in Pass 1, cache the result, and only
rebuild the address-dependent index structures in Pass 2. Expected
~2× speedup for chunked+deflate writes (512×512 deflate: 3.33ms → ~1.7ms).
2. SIMD-vectorisable shuffle filter (filters.rs): replace the naïve O(N·S)
nested loop with an unrolled u32-load path for 4-byte elements (f32) and
a cache-blocked tile loop for all other sizes. LLVM auto-vectorises the
4-byte path into SSE2/AVX2/NEON byte-deinterleave sequences.
3. Zstd benchmark variant (h5bench_write.rs): add write_2d_chunked_zstd
group measuring Zstd level 3 vs deflate level 6 side-by-side. Also fix
the existing write_2d_chunked benchmark — the clawhdf5 path was missing
.with_deflate(6), making the comparison apples-to-oranges. Add arXiv-
backed doc recommendation on DatasetBuilder::with_zstd().
4. Zero-copy HNSW save (hnsw.rs, clawhdf5-io/lib.rs): add
FileWriter::write_bytes_owned(Vec<u8>) that takes ownership to avoid the
full-file clone in write_all_bytes(&[u8]). HNSW::save_to_hdf5 uses it.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Critical fixes from whole-branch code review:
- libaec-sys: fix flag constants to match <libaec.h> exactly
(PREPROCESS=8, MSB=4, RESTRICTED=16; drop non-existent AEC_ALLOW_K13)
and add aec_buffer_encode FFI declaration
- filters_szip: fix cd index for bits_per_sample (cd[2] per H5Z_SZIP_PARM_BPP,
not cd[4]); fix option-mask mapping (NN=0x20, MSB unconditional); add two
real encode→decode roundtrip tests (no-NN and NN) that exercise libaec end-to-end
- file_writer: fix serialize_vds_mappings to delegate to data_layout_write
(eliminates the buggy duplicate that always emitted version=1 even for
external-file mappings); retains trailing Jenkins checksum
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
The previous aec_buffer_decode declaration used flat parameters which
don't match the actual libaec C API; this caused a SIGSEGV at runtime.
Replace with the correct aec_stream struct (mirroring <libaec.h>) and
update filters_szip.rs to populate and pass &mut AecStream.
Also add empty-input guard in szip_decode_impl and fallback library
path search in build.rs for distros that omit the .pc file.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Add `virtual_sources: Option<Vec<VdsMapping>>` field and `with_virtual_sources()` method to `DatasetBuilder`. In `FileWriter::finish()`, VDS datasets skip raw-data storage and instead serialize their source mappings into a global heap collection (version-1 same-file encoding) referenced by an HDF5 v4 layout-class-3 message. The two-pass address-computation loop handles VDS in both passes: pass 1 computes the fixed-size OH and pre-builds the heap blob; pass 2 places the blob at the correct file offset and rebuilds the OH with the real global heap address. Three new tests verify: (a) same-file two-source round-trip with mapping verification, (b) external-file source encoding, and (c) empty mapping list.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
clawhdf5-migrate:
- Real content validation: the post-migration check reads the written HDF5
back (new hdf5_reader) and compares actual content — chunk text, embeddings,
and every session/entity/relation field — to the source, not just row counts.
A representative sample of chunk rows is verified by default; --validate-full
checks every row. A count-preserving corruption no longer passes.
- Configurable schema: SQL is built from a SchemaConfig (table + ordered column
names, defaulting to the ZeroClaw layout) instead of hardcoded queries, with
--chunks-table / --sessions-table / --entities-table / --relations-table.
- Streaming count pass: --dry-run does a COUNT(*)-only pass per table instead
of loading every row.
- Incremental migration: --incremental reads the existing output, reads only
source chunks with id greater than the last migrated id, and appends them
(metadata groups refreshed from source) rather than re-migrating everything.
clawhdf5-format:
- read_as_f32 / read_as_f64 now decode IEEE-754 half-precision (2-byte) floats
via a no_std-safe bit conversion — needed to read float16-stored embeddings
back (e.g. for migrate's content validation), previously a TypeMismatch.
Tests: f16 read unit test; migrate tests for content-corruption detection,
custom table names, and incremental append; CLI smoke-tested end-to-end and the
dense/incremental output verified with h5py.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Dense attribute and dense link storage capped at a single fractal-heap direct
block (~64 KiB of heap data — a few thousand objects); beyond that the writer
produced an invalid oversized block. Lift the cap with a root indirect block.
When the serialized objects don't fit in one direct block, build a root
indirect block (FHIB) over multiple direct blocks sized by the doubling table
(start 512, width 4, doubling per row up to 64 KiB). Objects are packed
row-major across blocks, each block carries its logical block offset, and heap
IDs encode each object's heap offset (block offset + position). The FRHP points
root -> FHIB with the row count; unused slots in the current rows are undefined.
The fractal-heap builder is unified: FractalHeapBlock now carries the full heap
blob, and a shared write_frhp helper serializes the header for both the
single-block and multi-block paths. The single-block path is unchanged
(byte-identical), so existing dense attrs/links stay valid.
Validated end-to-end: a 2,500-attribute object and a 2,500-link group round-trip
through our reader and are read correctly by h5py. Objects still may not span a
block (no huge-object path).
Tests: facade round-trips for multi-block dense attrs and dense links, plus an
h5py-gated interop test (verified against the real h5py environment).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The fractal-heap reader split direct vs indirect block rows using the FRHP
"Starting # of Rows in Root Indirect Block" field (a constant, typically 1),
mislabeled as starting_row_of_indirect_blocks. For any heap whose data spans
more than one direct block — common in libhdf5 files with a large group or
many dense attributes — this treated direct blocks as indirect and walked into
garbage, failing with InvalidFractalHeapSignature.
Derive the split from the heap geometry instead: max_direct_rows =
log2(max_direct_block_size / starting_block_size) + 2. Rows below it hold
direct blocks; rows at/above hold child indirect blocks.
Validated against an h5py-written group with 400 dense attributes (root
indirect block, 4 rows, 13 direct blocks): all values now read correctly.
Regression fixture covers an 80-attribute multi-block heap.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
A group with more than 8 links (libhdf5's compact max_compact default) is now
written densely instead of as inline Link messages: the links live in a
single-direct-block fractal heap indexed by a v2 B-tree of type 5 (link-name
index), referenced from the group's LinkInfo message. This matches libhdf5's
compact->dense switchover and keeps large groups out of the object header.
- Extract the byte-identical fractal-heap builder shared by dense attributes
and dense links, parameterized by heap_id_length / max_heap_size. Attributes
keep 8 / 40; links use 7 / 32 to match libhdf5 (reverse-engineered: an
h5py-written dense group uses heap_id_length 7, max_heap_size 32, type-5
record = hash(4) + heap_id(7) = 11 bytes). This was the cause of an initial
"object overruns end of direct block" error from h5py.
- build_group_oh gained an optional dense LinkInfo (omitting inline Link
messages); the two-pass file assembly allocates each group's link blob after
its object header and rebuilds it with real target addresses in the final
pass (link-message size is address-independent, so layout is stable).
Validated end-to-end: our reader round-trips dense groups, h5py reads the
groups we write, and dense attributes remain byte-identical (still h5py-valid).
The agent's 9-dataset memory group now writes densely and round-trips. Single
direct block only (~a couple thousand links); indirect blocks remain a TODO.
Tests: facade round-trip (20-link dense group + compact sibling) and an
h5py-gated interop test confirming libhdf5 reads our dense groups.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
