Datatype::parse recurses into itself for Compound/Enumeration/
VariableLength/Array/Complex member and base types with no depth
counter. A message data size capped at u16::MAX (65535 bytes) allows
~8000 levels of nesting in a crafted file, enough to blow the stack —
worse on the project's no_std/embedded targets with only a few KB of
stack. Thread a depth counter through a new parse_with_depth, mirroring
object_header.rs's continuation-depth guard, and reject past 64 levels
with FormatError::NestingDepthExceeded. The public Datatype::parse
signature is unchanged.
INT-03
SymbolTableNode::parse used raw offset+8 arithmetic that can overflow
on a crafted v1-group B-tree leaf with a near-u64::MAX SNOD child
pointer (group_v1.rs passes such offsets through unchecked). Switch to
checked_add, matching read_offset in the same file. Also harden the
entries_start + num_symbols*entry_size computation with checked_add
for consistency, even though num_symbols being u16 already bounds
that multiply. Add regression tests.
INT-02
Six sites used raw `offset + N > file_data.len()` arithmetic that can
overflow on a crafted file with an address field near u64::MAX,
bypassing the bounds check before the next slice op panics. Switch to
the checked_add-based ensure_len pattern already used by local_heap.rs
and other parsers in this crate. Add regression tests for offsets near
usize::MAX in both files.
INT-01
Found via a real-world integration audit against omni-cortex (a JEPA-based
cognitive architecture built on clawhdf5 as its tiered Working/Episodic/
Semantic memory store).
- Add DatasetBuilder::with_u64_data (crates/clawhdf5-format/type_builders.rs).
The read side already has read_u64/read_as_u64, but there was no
symmetric write-side builder — only signed with_i32_data/with_i64_data
existed. Every consumer needing full-range u64 (timestamps, IDs) had to
bit-cast through i64 via `i64::from_ne_bytes(v.to_ne_bytes())` on write
and reverse it on read. omni-cortex does this in at least 6 places
across its writer/reader/mmap-reader/consolidate crates. Confirmed the
new builder round-trips full-range u64 (including values with the high
bit set) end-to-end in a standalone sanity check mirroring their usage.
- Fix Dataset::read_selection(&Selection::All) to route through the same
per-file chunk cache read_raw()/read_f64() etc. already use, instead of
the uncached read_chunked_data path. Selection::All is semantically a
full read; there's no reason two ways of asking for "everything" should
have different caching behavior. Also gains read_raw()'s virtual-dataset
resolver support for free. omni-cortex's Reader/mmap-reader/consolidate
crates all call read_selection(&Selection::All) for their chunked/
compressed dataset reads, so this was a real, if currently low-traffic
(single-pass read pattern), inconsistency in the public API's behavior.
- README: fix a stale crate-map claim that clawhdf5-filters supports
"blosc" compression — it never did (the crate only ever held
fast_deflate.rs; lz4/zstd/pcodec/szip filters live in clawhdf5-format).
New tests: u64_data_roundtrip, read_selection_all_matches_read_raw_on_chunked_dataset.
Add [workspace.dependencies] to the root Cargo.toml for the four
duplicated-across-many-crates dependencies flagged by the earlier review:
tempfile (7 crates), criterion (6), half (4 — real version skew, clawhdf5-gpu
pinned 2.7 while others used bare 2), and serde (4). Update every consuming
crate to `dep = { workspace = true }`, preserving crate-local `optional =
true` where it already existed. half now resolves uniformly to 2.7.x
workspace-wide instead of two separate semver ranges.
Also fixed clawhdf5-filters/Cargo.toml's stale "rustyhdf5" description
while touching the file (same class of leftover rename as prior fixes).
Not touching rayon/byteorder/clap (no skew found, lower priority).
- Add ensure_len(data, offset, needed) helper to chunked_read.rs,
data_read.rs, and local_heap.rs (matching the existing btree_v1.rs/
object_header.rs convention) and use it at every plain-arithmetic
offset+size bounds check found in these files, closing usize-overflow
panics reachable from crafted near-usize::MAX offsets/addresses.
- collect_chunk_info: add a depth-limited internal wrapper
(collect_chunk_info_inner, MAX_CHUNK_BTREE_DEPTH=64) to reject a
crafted self-referencing/cyclic B-tree v1 chunk index instead of
recursing unboundedly (stack-overflow DoS).
- read_compound_fields: validate byte_offset+field_size against the
compound's declared element size before slicing, instead of an
unguarded out-of-bounds panic on a crafted member offset.
- read_chunked_data/_cached/_sweep/_indexed: guard `ndims - 1` against
underflow for a degenerate zero-dimension chunked layout.
- copy_chunk_to_output: rewrite all offset/stride arithmetic (both the
1-D fast path and the general N-D path) to use checked_add/checked_mul,
skipping an out-of-range row/chunk instead of panicking on overflow.
Add a new cargo-fuzz target, fuzz_dataset_read, that walks every dataset
in a parsed file via the clawhdf5 facade and exercises the contiguous/
chunked/compact raw-data read paths that the existing fuzz_full_file
target doesn't reach. Seeded with the chunked/VDS/compound-relevant test
fixtures plus two crash regressions found during this pass (the
copy_chunk_to_output overflow and the ndims-1 underflow, both fixed
above — this target found real bugs within the first couple of runs).
Not wired into CI (nightly-only, multi-minute runs); documented in
fuzz/README.md as a manual/scheduled check instead. Also fixed the
README's stale rustyhdf5-format naming while touching this file.
Added regression tests for every fix (near-usize::MAX offsets, the
self-referencing B-tree case, the compound byte_offset overrun, the
zero-dim layout, and both copy_chunk_to_output overflow paths) so these
are caught by `cargo test`, not just the fuzz corpus.
- Fix version skew: clawhdf5-py (pyproject.toml 1.93.0 -> 2.1.0) and
packages/clawhdf5-node (package.json 2.0.0 -> 2.1.0) were both behind
the actual crate version.
- Correct stale ROADMAP.md claims: the TypeScript bridge already has a
complete napi-rs package (not "no package.json"); CI/CD is now wired
up via .gitea/workflows/ci.yml.
- Fix CLAUDE.md: clawhdf5-gpu uses wgpu with hand-written WGSL compute
shaders, not CubeCL.
- chunked_read.rs: drop 12 unnecessary chunk_dimensions[..rank].to_vec()
allocations — all three callees already accept &[u32].
- btree_v1.rs: add an overflow-safe ensure_len(data, offset, needed)
helper (checked_add) and use it at the two plain-arithmetic bounds
guards, closing a usize-overflow edge case reachable from a crafted
near-usize::MAX B-tree offset. Add a regression test.
- Clarify that the integrity hashes in clawhdf5-agent/provenance.rs
(FNV-1a) and clawhdf5-format/provenance.rs (SHA-256) are unkeyed and
only detect accidental corruption, not tampering — doc-only change.
- README.md: document that the mpi-io feature's read/write paths are
root-read+broadcast / gather-to-rank-0, not true collective I/O.
- Add .gitea/workflows/ci.yml running scripts/ci-test.sh (fmt, clippy,
test, no_std check) on push/PR to main.
- Fix stale rustyhdf5-py/rustyhdf5-format package names in
ci-test.sh/check-nostd.sh, which had been silently no-op'ing those
checks (cargo warns but doesn't fail on an unknown --exclude/-p
target).
- With those checks actually running, fix the real issues they surface:
- clippy: useless_conversion in chunked_write.rs, byte_char_slices in
global_heap.rs/object_header.rs.
- cargo fmt: apply formatting across the workspace (whitespace only).
- no_std (thumbv7em-none-eabihf) build errors in clawhdf5-format:
core::sync::atomic::AtomicU64 doesn't exist on that target (no
native 64-bit atomics) — switch profiling.rs's counters to
portable-atomic, which falls back to a CAS-based emulation there
and is a no-op wrapper elsewhere. Add missing alloc imports for
Box (filters.rs), Vec (filters_szip.rs), and format! (dict_encoding.rs)
on no_std paths. Replace f64::powi (std/libm-only) with a small
local exponentiation-by-squaring helper in the scale-offset filter.
The decompressed-chunk LRU cache was the hottest path in the read pipeline
(every chunked-dataset read goes through it) but did a linear scan through
up to 521 slots on every get/put, and a full buffer copy on every cache hit
(to_vec()/clone() of the whole decompressed chunk). chunked_read.rs then
cloned the buffer a second time just to insert it into the cache after
already having it in hand.
- Added a HashMap<ChunkCoord, usize> index alongside the LRU slots for O(1)
lookup. Eviction uses swap_remove, so the swapped-in slot's index entry is
fixed up on every eviction (covered by a dedicated test).
- CachedChunk.data is now Arc<CacheAlignedBuffer> — a cache hit is a
refcount bump, not a copy. CacheAlignedBuffer gained a Sync impl (same
soundness argument as its existing Send impl: access is only ever through
borrow-checked &/&mut, like Vec<u8>) so Arc<CacheAlignedBuffer> is itself
Send/Sync.
- put_decompressed/put_decompressed_aligned now return the Arc they just
inserted (or the existing cached copy), so callers can reuse that
allocation instead of holding a separate clone — eliminates the second
copy in chunked_read.rs's three call sites, which now consume the
Arc<CacheAlignedBuffer> (Deref's to &[u8], so downstream indexing/copy
code is unchanged).
- prefetch_hint's doc comment now leads with "bookkeeping only, does not
prefetch" instead of describing behavior it doesn't have.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
decompress_chunk() already threaded chunk_size (the pipeline's declared
decompressed size) into the scale-offset/nbit/szip decoders to bound their
output, but not into deflate/lz4/zstd/pcodec, all four of which allocated
based on attacker-controlled input with no cap:
- lz4: read a raw u32 "orig_size" straight from the compressed payload's
first 4 bytes and passed it directly to lz4_flex::block::decompress with
no upper bound — a 4-byte attacker-controlled field could request ~4 GiB.
- deflate (non-macOS path): unbounded flate2 read_to_end into a fresh Vec.
- zstd: zstd::decode_all with no output cap (classic decompression-bomb
vector, ratios can exceed 1000:1).
- pcodec: simple_decompress with no cap.
All four now take the expected chunk size and reject output that exceeds it
(or a 256 MiB absolute ceiling when the size is unavailable), matching the
pattern the other three filters already used. Also fixes the same unbounded
read_to_end in clawhdf5-filters' fast_deflate streaming fallback (used when
no size hint is available).
Added tests for each codec plus one exercising the actually-exploited path
through the public decompress_chunk() entrypoint.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
- Remove clawhdf5-types (empty 1-line stub crate; type defs already live in
clawhdf5-format). Update workspace Cargo.toml and CLAUDE.md accordingly.
- Implement HDF5 superblock v4 (page-buffer mode) read and write support in
clawhdf5-format: Superblock::parse_v4, page_size field, v4 serialize
branch, and FileWriter::with_page_size. This was the one task left
unimplemented from docs/superpowers/plans/2026-06-29-format-write-extensions.md.
- Reconcile the three docs/superpowers/plans/*.md docs (filter codecs,
format write extensions, MPI-IO VOL) against actual shipped code: they
were pre-work plans for d6c4d4f (2026-06-30) committed to git late on
2026-08-03 with all checkboxes still unchecked. Mark completed tasks done
and add a status note so they read as historical records, not open work.
- Refresh ROADMAP.md's "What's Next" section against current repo state.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Enables Rayon parallel compression for typical 4-chunk workloads (e.g.,
128×128 matrix with 32-row chunks). Rayon's dispatch overhead is ~2 µs,
worthwhile at ≥3 chunks with real compression work per chunk.
Previously the threshold was "> 4" which excluded 4-chunk datasets entirely
from parallel compression. Now "> 2" covers 3+ chunks.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Following arXiv:2506.18062 (TDT pre-filter) and matching h5py default
behavior: the shuffle filter is now automatically applied before any
compression codec (deflate, Zstd, LZ4, Pcodec) unless explicitly
disabled with .without_shuffle().
Benchmark results (f32 matrices, shuffle+codec vs unshuffled baseline):
- Zstd-3 at 512×512: 610 → 764 MiB/s (+25%)
- Deflate-6 at 128×128: 132 → 401 MiB/s (+204%)
- Deflate-6 at 512×512: 280 → 745 MiB/s (+166%)
Both codecs now reach parity at ~750 MiB/s for large matrices.
Changes:
- Add no_shuffle field to ChunkOptions (opt-out via .without_shuffle())
- Auto-add FILTER_SHUFFLE in build_pipeline() when compression is active
- Add DatasetBuilder.without_shuffle() method
- Update pipeline tests to reflect new 2-filter default
- Add chunk_options_pipeline_deflate_no_shuffle test
- Update BENCHMARKS.md with measured throughput improvements
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Implements Pcodec (filter ID 32023) via the `pco` 1.0.x crate as a new
optional compression codec. Pcodec achieves 30–94% better compression
ratio than Zstd for f32/f64 columnar data at 1–5 GiB/s decompression
speed, making it ideal for write-once/read-many embedding archives.
Write throughput at 512×512: 591 MiB/s (parity with Zstd-3 at 610 MiB/s).
For smaller chunks Zstd-3 remains faster due to Pcodec's fixed per-chunk
distributional analysis overhead.
- Add FILTER_PCODEC = 32023 constant to filter_pipeline.rs
- Add pcodec_compress/pcodec_decompress using pco::standalone API
- Wire into compress_chunk/decompress_chunk dispatch
- Add ChunkOptions.pcodec field and DatasetBuilder.with_pcodec() method
- Enable pcodec as highest-priority codec in build_pipeline()
- Add pco dep (optional, feature = "pcodec") to clawhdf5-format/clawhdf5
- Add write_2d_chunked_pcodec benchmark comparing pcodec vs zstd-3
- Document results in BENCHMARKS.md
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Four independent write-path improvements:
1. Cache compressed chunks between Pass 1 and Pass 2 (chunked_write.rs,
file_writer.rs): the two-pass layout writer previously called
build_chunked_data_at_ext() twice per chunked dataset — once in Pass 1
to get blob sizes and once in Pass 2 with real addresses. Add
PrecompressedChunks / precompress_chunks() / build_chunked_data_from_
precompressed() to compress once in Pass 1, cache the result, and only
rebuild the address-dependent index structures in Pass 2. Expected
~2× speedup for chunked+deflate writes (512×512 deflate: 3.33ms → ~1.7ms).
2. SIMD-vectorisable shuffle filter (filters.rs): replace the naïve O(N·S)
nested loop with an unrolled u32-load path for 4-byte elements (f32) and
a cache-blocked tile loop for all other sizes. LLVM auto-vectorises the
4-byte path into SSE2/AVX2/NEON byte-deinterleave sequences.
3. Zstd benchmark variant (h5bench_write.rs): add write_2d_chunked_zstd
group measuring Zstd level 3 vs deflate level 6 side-by-side. Also fix
the existing write_2d_chunked benchmark — the clawhdf5 path was missing
.with_deflate(6), making the comparison apples-to-oranges. Add arXiv-
backed doc recommendation on DatasetBuilder::with_zstd().
4. Zero-copy HNSW save (hnsw.rs, clawhdf5-io/lib.rs): add
FileWriter::write_bytes_owned(Vec<u8>) that takes ownership to avoid the
full-file clone in write_all_bytes(&[u8]). HNSW::save_to_hdf5 uses it.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Critical fixes from whole-branch code review:
- libaec-sys: fix flag constants to match <libaec.h> exactly
(PREPROCESS=8, MSB=4, RESTRICTED=16; drop non-existent AEC_ALLOW_K13)
and add aec_buffer_encode FFI declaration
- filters_szip: fix cd index for bits_per_sample (cd[2] per H5Z_SZIP_PARM_BPP,
not cd[4]); fix option-mask mapping (NN=0x20, MSB unconditional); add two
real encode→decode roundtrip tests (no-NN and NN) that exercise libaec end-to-end
- file_writer: fix serialize_vds_mappings to delegate to data_layout_write
(eliminates the buggy duplicate that always emitted version=1 even for
external-file mappings); retains trailing Jenkins checksum
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
The previous aec_buffer_decode declaration used flat parameters which
don't match the actual libaec C API; this caused a SIGSEGV at runtime.
Replace with the correct aec_stream struct (mirroring <libaec.h>) and
update filters_szip.rs to populate and pass &mut AecStream.
Also add empty-input guard in szip_decode_impl and fallback library
path search in build.rs for distros that omit the .pc file.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Add `virtual_sources: Option<Vec<VdsMapping>>` field and `with_virtual_sources()` method to `DatasetBuilder`. In `FileWriter::finish()`, VDS datasets skip raw-data storage and instead serialize their source mappings into a global heap collection (version-1 same-file encoding) referenced by an HDF5 v4 layout-class-3 message. The two-pass address-computation loop handles VDS in both passes: pass 1 computes the fixed-size OH and pre-builds the heap blob; pass 2 places the blob at the correct file offset and rebuilds the OH with the real global heap address. Three new tests verify: (a) same-file two-source round-trip with mapping verification, (b) external-file source encoding, and (c) empty mapping list.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
clawhdf5-migrate:
- Real content validation: the post-migration check reads the written HDF5
back (new hdf5_reader) and compares actual content — chunk text, embeddings,
and every session/entity/relation field — to the source, not just row counts.
A representative sample of chunk rows is verified by default; --validate-full
checks every row. A count-preserving corruption no longer passes.
- Configurable schema: SQL is built from a SchemaConfig (table + ordered column
names, defaulting to the ZeroClaw layout) instead of hardcoded queries, with
--chunks-table / --sessions-table / --entities-table / --relations-table.
- Streaming count pass: --dry-run does a COUNT(*)-only pass per table instead
of loading every row.
- Incremental migration: --incremental reads the existing output, reads only
source chunks with id greater than the last migrated id, and appends them
(metadata groups refreshed from source) rather than re-migrating everything.
clawhdf5-format:
- read_as_f32 / read_as_f64 now decode IEEE-754 half-precision (2-byte) floats
via a no_std-safe bit conversion — needed to read float16-stored embeddings
back (e.g. for migrate's content validation), previously a TypeMismatch.
Tests: f16 read unit test; migrate tests for content-corruption detection,
custom table names, and incremental append; CLI smoke-tested end-to-end and the
dense/incremental output verified with h5py.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Dense attribute and dense link storage capped at a single fractal-heap direct
block (~64 KiB of heap data — a few thousand objects); beyond that the writer
produced an invalid oversized block. Lift the cap with a root indirect block.
When the serialized objects don't fit in one direct block, build a root
indirect block (FHIB) over multiple direct blocks sized by the doubling table
(start 512, width 4, doubling per row up to 64 KiB). Objects are packed
row-major across blocks, each block carries its logical block offset, and heap
IDs encode each object's heap offset (block offset + position). The FRHP points
root -> FHIB with the row count; unused slots in the current rows are undefined.
The fractal-heap builder is unified: FractalHeapBlock now carries the full heap
blob, and a shared write_frhp helper serializes the header for both the
single-block and multi-block paths. The single-block path is unchanged
(byte-identical), so existing dense attrs/links stay valid.
Validated end-to-end: a 2,500-attribute object and a 2,500-link group round-trip
through our reader and are read correctly by h5py. Objects still may not span a
block (no huge-object path).
Tests: facade round-trips for multi-block dense attrs and dense links, plus an
h5py-gated interop test (verified against the real h5py environment).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The fractal-heap reader split direct vs indirect block rows using the FRHP
"Starting # of Rows in Root Indirect Block" field (a constant, typically 1),
mislabeled as starting_row_of_indirect_blocks. For any heap whose data spans
more than one direct block — common in libhdf5 files with a large group or
many dense attributes — this treated direct blocks as indirect and walked into
garbage, failing with InvalidFractalHeapSignature.
Derive the split from the heap geometry instead: max_direct_rows =
log2(max_direct_block_size / starting_block_size) + 2. Rows below it hold
direct blocks; rows at/above hold child indirect blocks.
Validated against an h5py-written group with 400 dense attributes (root
indirect block, 4 rows, 13 direct blocks): all values now read correctly.
Regression fixture covers an 80-attribute multi-block heap.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
A group with more than 8 links (libhdf5's compact max_compact default) is now
written densely instead of as inline Link messages: the links live in a
single-direct-block fractal heap indexed by a v2 B-tree of type 5 (link-name
index), referenced from the group's LinkInfo message. This matches libhdf5's
compact->dense switchover and keeps large groups out of the object header.
- Extract the byte-identical fractal-heap builder shared by dense attributes
and dense links, parameterized by heap_id_length / max_heap_size. Attributes
keep 8 / 40; links use 7 / 32 to match libhdf5 (reverse-engineered: an
h5py-written dense group uses heap_id_length 7, max_heap_size 32, type-5
record = hash(4) + heap_id(7) = 11 bytes). This was the cause of an initial
"object overruns end of direct block" error from h5py.
- build_group_oh gained an optional dense LinkInfo (omitting inline Link
messages); the two-pass file assembly allocates each group's link blob after
its object header and rebuilds it with real target addresses in the final
pass (link-message size is address-independent, so layout is stable).
Validated end-to-end: our reader round-trips dense groups, h5py reads the
groups we write, and dense attributes remain byte-identical (still h5py-valid).
The agent's 9-dataset memory group now writes densely and round-trips. Single
direct block only (~a couple thousand links); indirect blocks remain a TODO.
Tests: facade round-trip (20-link dense group + compact sibling) and an
h5py-gated interop test confirming libhdf5 reads our dense groups.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The readers added this cycle parse untrusted bytes, so malformed/hostile
input must produce errors — never a panic, OOM, or unbounded recursion.
Audited each new surface and fixed the concrete vectors, each covered by an
adversarial regression test:
- Paged Fixed Array: `1 << max_nelmts_bits` shift overflow (u8 up to 255);
element-count bounded by file size; element/page offset multiplies checked.
- H5S selection decoder: ALL/NONE validate they have the 16 bytes they claim
to consume; hyperslab rank capped at 32 (H5S_MAX_RANK); iter_linear
coordinate/stride/product arithmetic uses checked ops.
- VDS mapping parser: drop pre-allocation from the untrusted `nused`;
bounds-check all selection slicing.
- scale-offset / N-Bit filters: `1 << minbits` overflow at minbits==64; N-Bit
`bit_offset + precision` overflow; N-Bit type-tree recursion depth capped to
stop a crafted nested tree from overflowing the stack; element counts bounded
by the chunk's expected decompressed size (threaded the previously-unused
chunk_size into both decoders) so a bogus count can't over-allocate.
- VDS assembly: a virtual dataset whose source is itself virtual (a cycle) now
errors instead of recursing into a stack overflow.
16 new adversarial tests; full format suite (482 lib) + agent + facade green;
clippy clean.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
clawhdf5-agent: fixed-length string datasets (memory text chunks, session
summaries, ids, tags, entity/relation names) were stored uncompressed behind
a stale "chunked compound not yet supported" comment. Chunked writes work for
fixed-size string/compound datatypes like any other, so write_string_dataset
now chunks + deflates once a dataset's payload reaches 4 KiB — large,
redundant NullPad content compresses well while tiny metadata stays
contiguous (no chunk-overhead bloat). The dead `compress` parameter is
removed in favor of this size heuristic.
clawhdf5-format: enabling string compression exposed a latent bug — the
per-file ChunkCache built its chunk index once and reused it for every
chunked dataset in the file, keyed only by chunk coordinate with no dataset
discrimination. With one chunked dataset per file this never surfaced; with
two of different rank (a 1-D compressed string array and the 2-D embeddings
matrix) the first dataset's rank-1 index was reused for the second, panicking
with an out-of-bounds chunk coordinate. The cache now binds to a dataset by
its chunk-index address and rebinds — dropping the index, chunk-index map,
layout, and decompressed slots — whenever the dataset being read changes,
while still caching repeated/sequential access to the same dataset.
Tests: facade regression reading a 1-D compressed string dataset and a 2-D
compressed f32 dataset through one shared File cache (verified to panic
without the fix); existing agent e2e tests (large text chunks, migration
round-trip) now pass with compression on.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The HDF5 library does not implement the scale-offset filter's floating-point
E-scale mode. When asked for it (cd_values[0] = 1) it stores the chunk raw
(no minbits/minval header) and sets the chunk filter mask to skip the filter,
so such datasets read back verbatim purely by honoring the per-chunk filter
mask — no E-scale decoder is required.
Add a fixture written via the HDF5 low-level API (exact-representable values)
and a test asserting it reads back verbatim, locking in the filter-mask path.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
VDS sources living in other files were previously unsupported because the
pure-byte read API has no filesystem. Add a resolver seam and wire a default.
clawhdf5-format:
- Add VdsSourceResolver (Fn(&str) -> Option<Vec<u8>>) and
read_raw_data_full_with_resolver. read_virtual_data uses the resolver to
fetch an external source file's bytes by its stored name, then reads the
named source dataset from those bytes and scatters as usual. A resolver
returning None leaves the region at fill (HDF5's missing-source behavior);
an external source with no resolver at all is a clean error. read_raw_data_full
is unchanged (delegates with no resolver).
clawhdf5:
- File now records the directory it was opened from and, for virtual layouts,
reads through a default resolver that loads sibling source files relative to
that directory. So File::open(virt).dataset(d).read_*() transparently
assembles cross-file VDS. In-memory files (from_bytes) have no directory, so
only same-file VDS resolves there.
Tests: format-layer external read with an injected resolver (and the
no-resolver error path), plus facade tests that drop both files in a temp dir
and read through File::open — covering successful resolution and the
missing-source-is-fill case.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Generalize Selection iteration from 1-D to arbitrary rank: iter_linear(dims)
enumerates a selection's row-major linear indices over a dataspace of the
given shape (ALL, NONE, regular hyperslabs, points), which is the order HDF5
uses to pair virtual and source selections.
read_virtual_data now passes the full virtual/source dimensions instead of a
single extent, so multi-dimensional block mappings scatter to the correct
non-contiguous linear positions. read_named_dataset_raw returns the source
dataset's dimensions. The rank-1 restriction is removed; only external-file
sources remain unsupported.
Tests: 2-D integration fixture (vds_2d_same_file.h5: two 2x2 sources placed
as non-contiguous blocks in a 4x4 virtual) plus N-D iter_linear unit tests
(block, strided, ALL, rank-mismatch). The 1-D path is unchanged.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
A virtual layout previously returned UnsupportedVersion. Implement reading
for the common 1-D, same-file case, reverse-engineered and validated against
HDF5 2.0.
- Rewrite parse_vds_mappings to the real global-heap block format
(version(1) · nused(length_size) · entries · checksum(4)), where each
entry is source-file(null) · source-dataset(null) · source-selection ·
virtual-selection. Block version 1 encodes a same-file source as a single
0x04 marker in place of the file name; version 0 stores an explicit file
name. The selections are H5S-serialized and self-describing in length, so
they are decoded to find entry boundaries. The previous parser used a
guessed layout that did not match real files.
- Extend Selection with decode_serialized() (H5S_select_serialize: ALL,
NONE, and version-3 regular hyperslabs) and iter_linear_1d().
- Add read_virtual_data: resolve the mapping block from the global heap,
read each same-file source dataset, and scatter its selected elements into
the virtual buffer; unmapped regions stay at the zero fill value.
External-file sources and N-D selections return a clean unsupported error.
Tests: real-file integration test (vds_same_file.h5: partial source slice +
fill gap), selection decoder unit tests built from the fixture bytes, and
same-file/external mapping-parser unit tests.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
A filtered, fixed-dimension dataset with more than one Fixed Array
data-block page (>1024 chunks by default) previously failed with
"paged Fixed Array data blocks not yet supported".
Implement the paged data-block layout, reverse-engineered and validated
against an HDF5 2.0 file:
- after the FADB prefix: a page-init bitmap (one bit per page, MSB-first
within each byte), a 4-byte checksum, then the pages;
- each page is a fixed full-size slot of page_nelmts elements plus a
4-byte checksum, with only the final page shorter;
- uninitialized pages still occupy their slot (zero-filled), so the
bitmap — not a 0xFF sentinel — marks a whole page unallocated.
Element parsing is factored into parse_fa_element, shared by the
non-paged and paged paths.
Tests: real-file integration test against a minimal 2-page gzip fixture
(v4_fixed_array_paged.h5) plus a synthetic unit test covering a
multi-byte/MSB-first bitmap, a skipped uninitialized page, and a short
final page.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The typed read paths (read_as_i32/i64/u64/f32/f64) rejected Array datatypes
with a TypeMismatch, so an array-typed compound member (common with N-Bit /
reduced-precision data) could not be read. They now unwrap an Array to its base
type and read the flat sequence of base elements, recursing for nested arrays.
Base-type precision rules (e.g. reduced-precision sign extension) apply to the
elements.
Validated end-to-end against an HDF5 2.0 compound with an array member: the
array field reads [-1, 100, 1000, -32768] with correct 16-bit sign extension.
Adds a regression test for flat and nested array reads.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Generalizes the N-Bit filter (id 5) decoder from atomic-only to the full
recursive type tree carried in the filter client data: atomic
([1, size, order, precision, offset]), array ([2, total_size, <base>]) and
compound ([3, total_size, nmembers, (offset, <node>)*]), nestable to any depth.
The decoder parses the tree once, then walks it per element with an MSB-first
bit reader, placing each leaf field's significant bits at its byte/bit offset in
a zero-filled element — HDF5's canonical layout. Float members are encoded as
full-precision atomics and handled transparently. Validated end-to-end against
HDF5 2.0 / h5py: compound int+int, compound with an array member, and compound
with a float member all decode to the exact canonical bytes. Adds h5py-free unit
tests from real captured chunks. (Reading array-typed compound *fields* into a
flat buffer is a separate datatype-reader concern.)
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
HDF5 stores a fixed-point value whose datatype precision is smaller than its
storage size zero-filled above the precision; the sign of a reduced-precision
signed integer lives in the precision field, not the storage word, and is
applied during datatype conversion. clawhdf5 previously read the full storage
word, so e.g. a 16-bit-precision -1 (stored 0x0000ffff) read as 65535.
The integer read paths (read_as_i32/i64/u64/f32/f64) now extract the
[bit_offset, bit_offset+bit_precision) field and sign-extend (signed) or mask
(unsigned). Full-width types are unchanged — the bulk-copy fast paths are gated
to full width, so the common case keeps its memcpy and behaviour.
This completes signed N-Bit reads (now exact end-to-end) and also fixes
un-filtered reduced-precision signed/unsigned integer datasets. Validated
against HDF5 2.0 / h5py; adds h5py-free regression tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Implements decompression for the N-Bit filter (id 5), atomic integer/float
variant — previously returned UnsupportedFilter(5). N-Bit packs each element's
significant `precision` bits MSB-first with no header; decode reads those bits
per element and places them at the datatype's bit offset in a zero-filled
`size`-byte element, reproducing HDF5's canonical reduced-precision layout
(verified byte-for-byte against the equivalent un-filtered dataset).
Reverse-engineered and validated against HDF5 2.0 / h5py: unsigned
reduced-precision datasets now read end-to-end with exact values. Signed
reduced-precision values are restored to their canonical (zero-filled) bytes;
sign-extending them to the application width is the datatype reader's job — a
pre-existing concern shared with un-filtered reduced-precision data. Recursive
compound/array N-Bit layouts remain unsupported. Adds h5py-free unit tests from
real captured chunks.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Extends the scale-offset filter (id 6) decoder to the floating-point D-scale
variant (H5Z_SO_FLOAT_DSCALE) alongside the integer variant. Shares the header
parsing and MSB-first code unpacking; reconstruction is
`value = minval + code / 10^scale_factor`, where `minval` is the minimum float
stored in the header and the all-ones code is the (defined) fill value.
Reverse-engineered and validated against HDF5 2.0 / h5py across f32 and f64,
negatives, decimal scale factors D=1..5 and multi-chunk datasets (decoded values
match h5py to full precision). The float E-scale variant remains unsupported.
Adds h5py-free unit tests from real captured chunks.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Implements decompression for the scale-offset filter (id 6), integer mode
(H5Z_SO_INT) — previously returned UnsupportedFilter(6). The on-disk format was
reverse-engineered against HDF5 2.0 / h5py and verified across signed/unsigned
element sizes, negative minima, multi-chunk datasets and fill-value handling:
minbits (u32 LE) | 0x08 | minval (8 bytes LE) | 8 reserved bytes |
MSB-first packed codes (nelmts * minbits bits)
Each code is `value - minval`; the all-ones code is reserved for the (defined)
fill value. The floating-point variants (D-scale/E-scale) use a different
algorithm and remain reported as unsupported.
Validated end-to-end (a 200-element chunked scale-offset dataset, plus negative
and unsigned datasets, now decode to the exact h5py values). Adds h5py-free unit
tests using real captured compressed chunks.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Follow-up to the v5 compound fix, found by an interop sweep over diverse
h5py/HDF5 2.0 (libver=latest) datasets:
- Array datatype (class 10) version 5 was rejected. v3/v4/v5 share the same
array encoding, so the parser now accepts 3-5.
- Data Layout message version 5 was rejected, which broke EVERY chunked/
compressed dataset written by modern HDF5. v5 reuses the v4 message
structure, so it now routes through parse_v4.
Validated end-to-end: a gzip-compressed, Fixed-Array-indexed v5 chunked dataset
now decodes to the correct values. Adds h5py-free regression tests using the
real v5 array-datatype and chunked-layout bytes, and updates the layout
invalid-version test to use v6.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
clawhdf5 rejected datatype message version 5 for the compound class with
"invalid datatype version 5 for class 6", so it could not read compound
datasets written by modern HDF5 / h5py with libver=latest. v5 reuses the same
compact member encoding as v3/v4 (name, variable-width offset, member type), so
the parser now accepts versions 3-5 for compound.
Found by running the previously-ignored h5py interop tests against h5py 3.16 /
HDF5 2.0. Adds an h5py-free regression test using the real v5 datatype bytes.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
build_chunked_data_at_ext now compresses all chunks via compress_all_chunks
(previously dead code) before laying them out, so compression runs across
rayon threads under the `parallel` feature when there are >4 filtered chunks.
Layout is unchanged — compression preserves chunk order, so on-disk bytes are
identical to the sequential path. The agent crate enables `parallel`, so this
speeds up compressed embedding writes.
Removes the #[allow(dead_code)] on compress_all_chunks and gates
PARALLEL_COMPRESS_THRESHOLD behind the `parallel` feature.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Sweep of the docs after the v2.0.0 rename and recent changes:
- Per-crate READMEs (13 files): rename leftover rustyhdf5-*/edgehdf5-*
package names and badges to clawhdf5-*, bump usage versions to 2.1.0.
- README: update stale test badge (417 -> 1500+), workspace stats
(15 crates/72K -> 17 crates/84K), agent crate stats (20.7K/32 modules),
and add the missing clawhdf5-napi and clawhdf5-bench crates to the tree.
- CLAUDE.md: correct the CLI subcommand list (inspect/dump/index/search ->
the actual create/save/search/recall/stats/flush-wal/agents-md/export/snapshot).
No code changes. Verified there are zero todo!()/unimplemented!() macros and
no TODO/FIXME comments in the tree.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Bumps [workspace.package] and all 17 crate package versions (and internal
path-dependency requirements) from 2.0.0 to 2.1.0 for a coordinated release.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
ClawHDF5 Coding Agent