Covers performance, security, and provenance findings across
clawhdf5-format, clawhdf5-migrate, and memory/query crates. Each item
lists target file, problem, and proposed change for the coding phase.
than h5py (5e)
stable-worldmodel (arXiv 2605.21800, LeCun/Balestriero) supports HDF5 as
one of three native formats and measures generic HDF5 at 1,416-1,474
samples/s for per-frame sample loading. This measures clawhdf5 against
that shape, hardware-controlled: clawhdf5 and h5py reading the SAME file
on the SAME machine.
worldmodel_sampling example: mmap an (N,H,W,C) uint8 observation dataset,
read each frame once per pass in shuffled (dataloader) order. The file is
written by h5py (benchmarks/gen_worldmodel_frames.py) — clawhdf5 parsing
an externally-produced HDF5 file is itself the interop result — and read
by both clawhdf5 and the h5py counterpart (benchmarks/bench_worldmodel_h5py.py,
opening exactly stable-worldmodel's HDF5Dataset: swmr + 256 MB cache).
Results (tank, Ryzen 7 7800X3D, 20000x64x64x3 = 246 MB, in page cache,
median of 3):
clawhdf5 zero-copy view 593k samples/sec 8.1x
clawhdf5 materialised copy 518k samples/sec 7.1x
h5py (swmr, 256 MB cache) 73k samples/sec 1.0x
The materialised-copy row is the fair equal-work comparison (to_vec per
frame, matching h5py's numpy materialisation) and is still 7.1x faster;
that the copy costs almost nothing shows the gap is h5py's per-frame call
overhead, not data movement. Honest caveats in BENCHMARKS.md: absolute
numbers are NOT comparable to the paper's (different hardware, smaller
frames, no torch/transform), only the same-machine ratio is; this is an
in-page-cache measurement isolating read-path overhead, not disk
bandwidth.
Adds only an example, two benchmark scripts, and a BENCHMARKS.md section —
no library code. (Workspace clippy has pre-existing toolchain drift
unrelated to this change; tracked separately.)
Tier 4b reported hybrid retrieval at 0.7/0.3 and noted the weights were "the
documented default, not a searched optimum". `--sweep` searches them: 0.0 to 1.0
in 0.1 steps, reusing the one-time embedding table so eleven configurations cost
barely more than three.
The result is not a refinement. 0.7/0.3 is **strictly dominated**:
vector/keyword Hit@1 Hit@5 Hit@10 MRR sHit@5
0.0 / 1.0 53.8% 75.0% 81.6% 0.6320 93.6%
0.3 / 0.7 53.2% 78.8% 87.2% 0.6463 96.0%
0.4 / 0.6 51.6% 81.4% 87.8% 0.6429 96.8%
0.5 / 0.5 48.2% 81.4% 88.2% 0.6234 97.4%
0.7 / 0.3 44.4% 79.2% 86.0% 0.5868 95.8%
1.0 / 0.0 36.0% 71.8% 81.6% 0.5027 94.2%
0.4/0.6 beats 0.7/0.3 on every metric at both granularities — Hit@1 +7.2pp,
Hit@5 +2.2, Hit@10 +1.8, MRR +0.056. No trade is being made; the default simply
sat on the wrong side of the peak. It is now 0.4/0.6, and README's usage snippet
recommends the same.
This corrects a conclusion I published one commit ago. Measuring only 0.7/0.3, I
wrote that fusion "buys deeper recall and pays for it at rank 1" and advised
callers taking a single top hit to prefer BM25. That was an artifact of the bad
weight, not a property of fusion: at 0.3/0.7 hybrid *beats* BM25 on MRR (0.6463
vs 0.6320) and Hit@5 (78.8% vs 75.0%) while giving up 0.6pp of Hit@1. Both
BENCHMARKS.md and README carry the correction rather than a quiet edit, since
the old text told readers to configure their systems a particular way.
The three-mode ablation rows are kept at their original settings — they measure
the shape of each stage in isolation, and the operating point now comes from the
sweep instead.
The GPU path worked but was effectively hidden. cudarc's build script shells out
to `nvcc`, which ships in /usr/local/cuda/bin — a directory the reference host
had installed but never exported to the login shell, so `--features
embeddings-cuda` failed with a bare "`nvcc --version` failed" panic from a
dependency's build script, and the runtime fallback then reported only
"Embedder: CPU (...)" before spending hours on work a GPU does in minutes.
Two changes, both about making the failure legible rather than changing what the
code does:
- The CPU fallback now says why it fell back and what that costs, with the
concrete fix. A run that silently takes two orders of magnitude longer reads
as a hang, not as a configuration choice.
- BENCHMARKS.md states the build-time nvcc requirement, where the toolkit
actually installs, and that a shell file read non-interactively is the place
to export it — `~/.zshenv` rather than `~/.zshrc`, because build scripts do
not run in an interactive shell.
Host-side, the reference machine's CUDA exports lived in ~/.bashrc below its
non-interactive guard while the login shell is zsh, so they never applied to
anything. Moved to ~/.zshenv with duplicate-prepend guards; `nvcc --version`
and `cargo build --features embeddings-cuda` now both work over a plain
non-interactive ssh with no manual export.
Every LongMemEval number this project has published measured BM25 alone. The
bench passed zero-vector embeddings with vector_weight=0.0, so the HNSW/vector
stage — the thing the README credits for retrieval quality — contributed
nothing and was never tested.
An optional `embeddings` feature loads all-MiniLM-L6-v2 via candle and encodes
the corpus for real. It is off by default and nothing in the shipped crates
depends on it, so a project that advertises no heavyweight dependencies keeps
that property; without the feature the bench behaves exactly as before.
Full haystack, n=500, turn-level:
Hit@1 Hit@5 Hit@10 MRR
BM25 only 53.8% 75.0% 81.6% 0.6320
Vector only 36.0% 71.8% 81.6% 0.5027
Hybrid 0.7/0.3 44.4% 79.2% 86.0% 0.5868
Session-level, hybrid leads outright: 88.2 / 95.8 / 97.8 / 0.9158.
The hybrid claim holds for depth and not for precision@1. Hybrid is the best
configuration at Hit@5 and Hit@10 at both granularities — turn-level Hit@5 gains
4.2 points over BM25 and 7.4 over vector-only, which is the result that justifies
running two stages at all. But BM25 alone still leads turn-level Hit@1 and MRR,
so fusing buys deeper recall and pays at rank 1. Callers assembling five memories
of context want hybrid; callers taking a single top hit are better served by BM25
today. The 0.7/0.3 weights are the documented default, not a searched optimum.
omni-cortex's four-signal ablation found the same direction independently — there,
adding BM25 to a dense retriever raised nDCG@5 while lowering Hit@1 and MRR. Two
codebases, two fusion schemes, same trade.
Vector-only trailing BM25 at every turn-level cutoff except Hit@10 is stated
plainly rather than buried: LongMemEval questions share heavy vocabulary with
their evidence turns, which is close to the best case for lexical matching, and
MiniLM at 384-d is a small model.
Implementation notes:
- Texts are deduplicated before encoding. The haystack sessions are drawn from
a shared pool, so 500 questions x 493.5 turns collapses to 190,015 unique
strings — the difference between encoding the corpus once and per question.
- `embeddings-cuda` adds the GPU path, and it is not a convenience: 190k texts
take ~13 min on an RTX 5060 Ti, while the same work on 8 CPU cores was still
unfinished after 30 minutes. The device is selected at runtime with a CPU
fallback, so a machine without CUDA still works.
- Mean-pooling is masked and the output L2-normalised, which is the published
recipe for this checkpoint (not the [CLS] pooler).
One measurement wrinkle, recorded rather than smoothed over: on the oracle
variant BM25-only reads 84.2% Hit@5 with real embedding vectors present against
84.4% with zero vectors — one question of 500 changes rank, MRR identical at
0.6597. On the full haystack the two agree exactly. Weight 0.0 evidently does not
make the vector stage bit-for-bit absent from candidate selection on a small
corpus.
Verified on the Linux dev host: 49 groups / 1659 passed / 0 failed, clippy clean
under -D warnings, fmt clean, with and without the feature.
The harness only ever ran longmemeval_oracle — evidence sessions only, which is
a substantially easier corpus than the dataset LongMemEval results are normally
quoted on. Worse, the variant was a hardcoded "oracle" string in both the report
header and the JSON summary, so pointing it at longmemeval_s would have produced
full-haystack numbers labelled oracle.
DatasetProfile now measures the corpus instead of asserting it: sessions and
turns per question, and evidence-session density (the mean share of a question's
haystack sessions that are answer sessions). The variant label and the
session-level degeneracy warning are both derived from that density, so a
mislabelled input file cannot produce a mislabelled result. Measured: 100.0%
density on the oracle variant, 4.0% on longmemeval_s.
The full haystack, all 500 questions, 47.7 sessions and 493.5 turns each:
turn-level session-level
Hit@1 53.8% 86.2%
Hit@5 75.0% 93.6%
Hit@10 81.6% 96.6%
MRR 0.6320 0.8948
Turn-level drops 84.4% -> 75.0% against the oracle variant. That 9.4-point gap
is the price of the real haystack and is exactly why oracle-only numbers should
not be presented as LongMemEval results.
Session-level is now reportable. It was retracted before because at 100% evidence
density every returned document is a hit by construction; at 4.0% density a hit
reflects discrimination, so 93.6% is a real measurement rather than a restatement
of the corpus shape. Per-type it also finally separates: single-session-assistant
100.0% Hit@1 against single-session-preference 33.3% — BM25 has nothing to grip
on a preference question whose evidence shares no vocabulary with the query.
The MemX comparison stays withdrawn. Running the full haystack closes the corpus
half of that mismatch but not the granularity half: MemX measures fact-level over
220,349 records, and this harness measures turn- and session-level.
Two smaller fixes found while running it:
- --limit samples evenly across the file rather than taking a prefix. The
dataset is ordered by question type, so `--limit 20` returned 20
single-session-user questions and nothing else while reading like a
whole-dataset result.
- abstention_accuracy emits null rather than 0.0 when a corpus poses no
abstention questions. longmemeval_s has none, and 0.0000 reads as total
failure at a task that was never asked.
README.md and BENCHMARKS.md now lead with the full-haystack numbers and keep the
oracle figures alongside, labelled as the easier corpus.
Verified on the Linux dev host: 49 groups / 1659 passed / 0 failed, clippy clean
under -D warnings, fmt clean. The full 500-question run takes ~70 s.
A methodology audit found that two benchmark claims published in this repo two
days ago measure the wrong thing. Both are retracted in place rather than
quietly edited, with the reasoning recorded.
1. Session-level LongMemEval recall (100.0% Hit@1/5/10, MRR 1.0000, uniform
across all six question types) is a degenerate artifact. On the
longmemeval_oracle variant the ingested haystack for a question is
essentially only that question's evidence sessions, so every returned
document belongs to an answer session and session-level hit rate is ~1.0 at
rank 0 by construction. The uniform 100% across every question type was the
tell. It measured the shape of the corpus, not the retriever. Only the
turn-level figure (84.4% Hit@5) carries signal, and it is now the only
retrieval number cited.
2. The "clawhdf5 outperforms MemX at turn-level retrieval (84.4% vs 51.6%)"
claim was not like-for-like on two independent axes. Confirmed against
arxiv:2603.16171: MemX's Hit@5=51.6% / MRR=0.380 is *fact-level*
granularity over 220,349 fact-level records drawn from 19,195 sessions, and
the paper explicitly notes fact-level "doubl[es] session-level performance".
Ours is turn-level on the oracle subset — different granularity, and a
corpus smaller by orders of magnitude. A higher number on an easier corpus
at a different granularity is not an outperformance claim.
Also caveats the vector-search "vs MemX" latency ratios, which compare a single
clawhdf5 component (raw vector search) against MemX's end-to-end pipeline
figure (embeddings + FTS5 + four-factor re-ranking). The numbers are real; the
"speedup" framing overstated by an unquantified margin and is now labelled an
order-of-magnitude indication.
Adds an explicit scoring-target declaration to BENCHMARKS.md per arXiv
2605.24060, which found that changing scoring target alone alters nDCG on
83-94% of queries and can reverse system rankings. States dataset variant,
metric (retrieval recall, NOT the official QA-accuracy metric), granularity,
k, and that the vector stage is inert (zero embeddings, vector_weight=0.0).
The harness itself now prints its scoring target, flags the session-level
block as degenerate, warns against the MemX comparison, and emits
dataset_variant/scoring_target/k/session_level_degenerate in its JSON summary,
so the caveats travel with the numbers instead of living only in docs.
Found via a real-world integration audit against omni-cortex (a JEPA-based
cognitive architecture built on clawhdf5 as its tiered Working/Episodic/
Semantic memory store).
- Add DatasetBuilder::with_u64_data (crates/clawhdf5-format/type_builders.rs).
The read side already has read_u64/read_as_u64, but there was no
symmetric write-side builder — only signed with_i32_data/with_i64_data
existed. Every consumer needing full-range u64 (timestamps, IDs) had to
bit-cast through i64 via `i64::from_ne_bytes(v.to_ne_bytes())` on write
and reverse it on read. omni-cortex does this in at least 6 places
across its writer/reader/mmap-reader/consolidate crates. Confirmed the
new builder round-trips full-range u64 (including values with the high
bit set) end-to-end in a standalone sanity check mirroring their usage.
- Fix Dataset::read_selection(&Selection::All) to route through the same
per-file chunk cache read_raw()/read_f64() etc. already use, instead of
the uncached read_chunked_data path. Selection::All is semantically a
full read; there's no reason two ways of asking for "everything" should
have different caching behavior. Also gains read_raw()'s virtual-dataset
resolver support for free. omni-cortex's Reader/mmap-reader/consolidate
crates all call read_selection(&Selection::All) for their chunked/
compressed dataset reads, so this was a real, if currently low-traffic
(single-pass read pattern), inconsistency in the public API's behavior.
- README: fix a stale crate-map claim that clawhdf5-filters supports
"blosc" compression — it never did (the crate only ever held
fast_deflate.rs; lz4/zstd/pcodec/szip filters live in clawhdf5-format).
New tests: u64_data_roundtrip, read_selection_all_matches_read_raw_on_chunked_dataset.
README.md:
- Fix badly stale LongMemEval numbers (badge said Hit@5 46%, table showed
fabricated ~46%/~0.34/~72% figures that never matched BENCHMARKS.md's
actual results of Hit@5 100% session / 84.4% turn-level, MRR 1.0/0.6597)
- Remove clawhdf5-types from the Crate Map — that crate was removed in an
earlier cleanup pass but the README diagram was never updated; fix the
crate count (16, not 17) and stale line-of-code figures (72,087/84K -> ~92K)
- Fix a dead #benchmarks badge anchor (no such heading exists) -> #performance
- Document the new clawhdf5-ann `parallel` feature (had no Feature Flags entry)
- Note WAL's CRC32 per-entry check, link the new tank LongMemEval/SIMD/
vector-search reproduction section, update stale test-count comment
(417+ -> 1,650+) and Phase 2 roadmap blurb (LongMemEval is now done)
ROADMAP.md:
- Check off "Academic benchmark cross-validation" (done via the tank
LongMemEval re-run) and add a new "Recently closed out" section
summarizing the Tier 3-4 hardening pass (Android JNI validation, pyo3
bump, WAL CRC32, bounds-check audit + fuzz harness that found 3 real
bugs, HNSW optional parallel feature, workspace.dependencies)
- Update stale test count (1,546 -> 1,650+) and last-updated date
CLAUDE.md: mention WAL's per-entry CRC32 check
CHANGELOG.md: add Security/Performance/Architecture/Documentation entries
under Unreleased summarizing all of Tiers 1-4 (this had not been touched
since 2026-06-04, predating the entire hardening pass)
Re-ran the three previously-undated sections flagged by the top-of-file
traceability note on tank (Ryzen 7 7800X3D, 2026-08-05), the same machine
already used for the vs-libhdf5 validation:
- LongMemEval Results: recall numbers reproduce exactly (deterministic
BM25 retrieval), latency numbers are new/hardware-specific and higher
than the i7 citation with much wider variance — recorded as-is.
- SIMD & Parallelism: found that several of the originally-named
benchmarks don't actually hold the dataset fixed while varying only
the SIMD/scalar/parallel axis — several call the same underlying
function under different names. Used adaptive_benches' strategy_*
benchmarks instead, which genuinely do isolate that axis via the
SearchStrategy enum. Real finding: the speedup on tank (~1.5x) is
smaller than on the i7 (~2.0x), attributed to the Ryzen's large L3
cache narrowing the scalar-vs-SIMD gap — recorded rather than
reconciled away.
- Vector Search Latency / Comparison to MemX: re-run with tank numbers,
all faster than the i7 citation as expected; the 1K Pre-norm cell has
no corresponding benchmark in the current suite and is left blank
rather than guessed.
Updated the top-of-file traceability note to reflect that these three
sections (plus Comparison to MemX) now meet the dated/hardware-cited/
reproducible bar, narrowing the list of sections that don't.
Bump WAL_VERSION to 2: every entry (Save and Tombstone) now ends with a
4-byte CRC32 trailer computed over its type+timestamp+payload bytes, using
the existing clawhdf5_format::checksum::crc32 (already available since
clawhdf5-agent depends on clawhdf5-format with fast-checksum enabled).
A bit-flip inside an entry is now detected and replay stops there, instead
of silently accepting corrupted data as before.
Write side needed no restructuring — append_save/append_tombstone already
buffer an entry's bytes before a single write_all, so the CRC is just
appended to that buffer first.
Read side: read_len_prefixed_str/read_embedding are generalized from
&mut File to R: Read, and a new TeeReader<R> wraps the file handle for one
entry at a time, accumulating every byte actually consumed (via read_exact)
into a buffer. This lets read_entries compute the CRC over exactly the
bytes read for a Save entry without needing to know its length up front
(its sub-fields are length-prefixed and interleaved with the length itself
only becoming known as parsing proceeds). A new read_one_entry<R: Read>
factors the per-entry-type field parsing shared by both the legacy and
current read paths.
Backward compatibility: WAL_VERSION_LEGACY_NO_CRC (1) files are still
readable via WalFile::read_entries (old field-by-file-handle path,
unchanged, no CRC expected). WalFile::open migrates a legacy file by
recreating it fresh in the current format — safe because the only two
real call sites (HDF5Memory::open/create) always call read_entries before
open, so entries are already replayed by the time migration happens.
New tests: a corrupted-payload-byte test confirming replay stops cleanly
at the corrupted entry (no prior coverage existed for mid-entry bit-flip
detection), a legacy-v1-format read test, and an open()-migration test.
Add a default-off `parallel` feature to clawhdf5-ann (rayon optional dep),
matching the convention already used in clawhdf5-format/clawhdf5-agent.
Gate prune_connections' per-neighbor distance computation on it — a pure
read-only map with no shared mutable state, sorted immediately after, so
swapping to rayon's par_iter is low-risk.
Deliberately not touching build_with_metric's outer insert loop per the
original plan: it has genuine cross-iteration data dependencies (graph
mutation, entry-point updates) and needs its own correctness-focused
design pass. The win here is likely small since neighbor lists are
bounded by m/m_max0 (typically small) — this is a low-risk completeness
item, not a headline perf change.
Verified identical results with default features and --features parallel
across the full HNSW test suite (23/23 both ways), including the
build+search end-to-end tests (build_small_index, search_accuracy_cosine,
incremental_insert_matches_batch_recall).
Add [workspace.dependencies] to the root Cargo.toml for the four
duplicated-across-many-crates dependencies flagged by the earlier review:
tempfile (7 crates), criterion (6), half (4 — real version skew, clawhdf5-gpu
pinned 2.7 while others used bare 2), and serde (4). Update every consuming
crate to `dep = { workspace = true }`, preserving crate-local `optional =
true` where it already existed. half now resolves uniformly to 2.7.x
workspace-wide instead of two separate semver ranges.
Also fixed clawhdf5-filters/Cargo.toml's stale "rustyhdf5" description
while touching the file (same class of leftover rename as prior fixes).
Not touching rayon/byteorder/clap (no skew found, lower priority).
- Add ensure_len(data, offset, needed) helper to chunked_read.rs,
data_read.rs, and local_heap.rs (matching the existing btree_v1.rs/
object_header.rs convention) and use it at every plain-arithmetic
offset+size bounds check found in these files, closing usize-overflow
panics reachable from crafted near-usize::MAX offsets/addresses.
- collect_chunk_info: add a depth-limited internal wrapper
(collect_chunk_info_inner, MAX_CHUNK_BTREE_DEPTH=64) to reject a
crafted self-referencing/cyclic B-tree v1 chunk index instead of
recursing unboundedly (stack-overflow DoS).
- read_compound_fields: validate byte_offset+field_size against the
compound's declared element size before slicing, instead of an
unguarded out-of-bounds panic on a crafted member offset.
- read_chunked_data/_cached/_sweep/_indexed: guard `ndims - 1` against
underflow for a degenerate zero-dimension chunked layout.
- copy_chunk_to_output: rewrite all offset/stride arithmetic (both the
1-D fast path and the general N-D path) to use checked_add/checked_mul,
skipping an out-of-range row/chunk instead of panicking on overflow.
Add a new cargo-fuzz target, fuzz_dataset_read, that walks every dataset
in a parsed file via the clawhdf5 facade and exercises the contiguous/
chunked/compact raw-data read paths that the existing fuzz_full_file
target doesn't reach. Seeded with the chunked/VDS/compound-relevant test
fixtures plus two crash regressions found during this pass (the
copy_chunk_to_output overflow and the ndims-1 underflow, both fixed
above — this target found real bugs within the first couple of runs).
Not wired into CI (nightly-only, multi-minute runs); documented in
fuzz/README.md as a manual/scheduled check instead. Also fixed the
README's stale rustyhdf5-format naming while touching this file.
Added regression tests for every fix (near-usize::MAX offsets, the
self-referencing B-tree case, the compound byte_offset overrun, the
zero-dim layout, and both copy_chunk_to_output overflow paths) so these
are caught by `cargo test`, not just the fuzz corpus.
- clawhdf5-android: validate embedding_len/query_embedding_len against
the handle's configured embedding_dim (and reject null pointers)
before constructing a slice via from_raw_parts in edgehdf5_save and
edgehdf5_hybrid_search. Strengthen the # Safety docs to state the
now-enforced invariant and its limits. Add unit tests covering
mismatched length and null-pointer rejection.
- clawhdf5-py: bump pyo3/numpy 0.28 -> 0.29, clearing RUSTSEC-2026-0176
(OOB read in PyList/PyTuple iterator) and RUSTSEC-2026-0177 (missing
Sync bound on PyCFunction::new_closure). No source changes needed;
confirmed via cargo audit that both advisories no longer appear.
- clawhdf5-agent/wal.rs: cap read_len_prefixed_str/read_embedding's
length claims at a new MAX_WAL_FIELD_LEN (64 MiB) before allocating,
so a corrupted/truncated WAL length field fails cleanly instead of
attempting a huge allocation. Add regression tests for both.
- BENCHMARKS.md: add a top-of-file traceability note distinguishing the
dated/hardware-cited/reproducible h5bench and tank-validation sections
from the older sections that don't yet meet that bar.
- Fix version skew: clawhdf5-py (pyproject.toml 1.93.0 -> 2.1.0) and
packages/clawhdf5-node (package.json 2.0.0 -> 2.1.0) were both behind
the actual crate version.
- Correct stale ROADMAP.md claims: the TypeScript bridge already has a
complete napi-rs package (not "no package.json"); CI/CD is now wired
up via .gitea/workflows/ci.yml.
- Fix CLAUDE.md: clawhdf5-gpu uses wgpu with hand-written WGSL compute
shaders, not CubeCL.
- chunked_read.rs: drop 12 unnecessary chunk_dimensions[..rank].to_vec()
allocations — all three callees already accept &[u32].
- btree_v1.rs: add an overflow-safe ensure_len(data, offset, needed)
helper (checked_add) and use it at the two plain-arithmetic bounds
guards, closing a usize-overflow edge case reachable from a crafted
near-usize::MAX B-tree offset. Add a regression test.
- Clarify that the integrity hashes in clawhdf5-agent/provenance.rs
(FNV-1a) and clawhdf5-format/provenance.rs (SHA-256) are unkeyed and
only detect accidental corruption, not tampering — doc-only change.
- README.md: document that the mpi-io feature's read/write paths are
root-read+broadcast / gather-to-rank-0, not true collective I/O.
- Add .gitea/workflows/ci.yml running scripts/ci-test.sh (fmt, clippy,
test, no_std check) on push/PR to main.
- Fix stale rustyhdf5-py/rustyhdf5-format package names in
ci-test.sh/check-nostd.sh, which had been silently no-op'ing those
checks (cargo warns but doesn't fail on an unknown --exclude/-p
target).
- With those checks actually running, fix the real issues they surface:
- clippy: useless_conversion in chunked_write.rs, byte_char_slices in
global_heap.rs/object_header.rs.
- cargo fmt: apply formatting across the workspace (whitespace only).
- no_std (thumbv7em-none-eabihf) build errors in clawhdf5-format:
core::sync::atomic::AtomicU64 doesn't exist on that target (no
native 64-bit atomics) — switch profiling.rs's counters to
portable-atomic, which falls back to a CAS-based emulation there
and is a no-op wrapper elsewhere. Add missing alloc imports for
Box (filters.rs), Vec (filters_szip.rs), and format! (dict_encoding.rs)
on no_std paths. Replace f64::powi (std/libm-only) with a small
local exponentiation-by-squaring helper in the scale-offset filter.
The decompressed-chunk LRU cache was the hottest path in the read pipeline
(every chunked-dataset read goes through it) but did a linear scan through
up to 521 slots on every get/put, and a full buffer copy on every cache hit
(to_vec()/clone() of the whole decompressed chunk). chunked_read.rs then
cloned the buffer a second time just to insert it into the cache after
already having it in hand.
- Added a HashMap<ChunkCoord, usize> index alongside the LRU slots for O(1)
lookup. Eviction uses swap_remove, so the swapped-in slot's index entry is
fixed up on every eviction (covered by a dedicated test).
- CachedChunk.data is now Arc<CacheAlignedBuffer> — a cache hit is a
refcount bump, not a copy. CacheAlignedBuffer gained a Sync impl (same
soundness argument as its existing Send impl: access is only ever through
borrow-checked &/&mut, like Vec<u8>) so Arc<CacheAlignedBuffer> is itself
Send/Sync.
- put_decompressed/put_decompressed_aligned now return the Arc they just
inserted (or the existing cached copy), so callers can reuse that
allocation instead of holding a separate clone — eliminates the second
copy in chunked_read.rs's three call sites, which now consume the
Arc<CacheAlignedBuffer> (Deref's to &[u8], so downstream indexing/copy
code is unchanged).
- prefetch_hint's doc comment now leads with "bookkeeping only, does not
prefetch" instead of describing behavior it doesn't have.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
decompress_chunk() already threaded chunk_size (the pipeline's declared
decompressed size) into the scale-offset/nbit/szip decoders to bound their
output, but not into deflate/lz4/zstd/pcodec, all four of which allocated
based on attacker-controlled input with no cap:
- lz4: read a raw u32 "orig_size" straight from the compressed payload's
first 4 bytes and passed it directly to lz4_flex::block::decompress with
no upper bound — a 4-byte attacker-controlled field could request ~4 GiB.
- deflate (non-macOS path): unbounded flate2 read_to_end into a fresh Vec.
- zstd: zstd::decode_all with no output cap (classic decompression-bomb
vector, ratios can exceed 1000:1).
- pcodec: simple_decompress with no cap.
All four now take the expected chunk size and reject output that exceeds it
(or a 256 MiB absolute ceiling when the size is unavailable), matching the
pattern the other three filters already used. Also fixes the same unbounded
read_to_end in clawhdf5-filters' fast_deflate streaming fallback (used when
no size hint is available).
Added tests for each codec plus one exercising the actually-exploited path
through the public decompress_chunk() entrypoint.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
- README's "HDF5 Core I/O" table claimed 19ns/2,080µs labeled 308× (real ratio
~109,000×) and a 313ns zero-copy mmap figure — neither traced to any dated
benchmark in BENCHMARKS.md. Replaced the table wholesale with the existing
"vs libhdf5 Summary" figures, relabeled from "h5py/C HDF5" to "libhdf5"
(BENCHMARKS.md never benchmarks against h5py, only libhdf5 directly).
- Added two new Criterion benchmarks to close the coverage gaps that produced
the untraceable numbers: metadata_open_from_disk (I/O-inclusive, fair
clawhdf5-vs-libhdf5 file-open comparison) and metadata_parse_in_memory
(clawhdf5-only, explicitly labeled as excluding I/O) in h5bench_meta.rs;
read_zerocopy_mmap in h5bench_read.rs (forces real page-ins by summing
elements rather than just returning a slice length — the mmap path turns
out to be slower than a plain copy at these sizes, an honest, unflattering
but real result now documented instead of a fabricated 313ns).
- Re-ran the full existing benchmark suite plus the two new ones on a second,
independently administered machine (tank: Ryzen 7 7800X3D) to validate the
numbers before publishing them. 5 of 6 rows landed within ~15% of the
original i7-12650H figures; recorded both in BENCHMARKS.md's new
"Independent Validation" section. README now cites the tank numbers.
- Added a short top-of-file README callout naming both halves of the project
(general-purpose HDF5 library vs. agent memory layer) with links to
BENCHMARKS.md and the Crate Map, so a data-infra reader isn't 60% through
a memory-store pitch before finding the part relevant to them.
- Added one factual, no-names line noting benchmark numbers are being
validated in collaboration with HDF5 Group engineers.
- Fixed the same untraceable "2-300x faster than h5py/C HDF5" / "313 ns"
claims in docs/QUICKSTART.md, one click from the README's own "New here?"
link.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
- Remove clawhdf5-types (empty 1-line stub crate; type defs already live in
clawhdf5-format). Update workspace Cargo.toml and CLAUDE.md accordingly.
- Implement HDF5 superblock v4 (page-buffer mode) read and write support in
clawhdf5-format: Superblock::parse_v4, page_size field, v4 serialize
branch, and FileWriter::with_page_size. This was the one task left
unimplemented from docs/superpowers/plans/2026-06-29-format-write-extensions.md.
- Reconcile the three docs/superpowers/plans/*.md docs (filter codecs,
format write extensions, MPI-IO VOL) against actual shipped code: they
were pre-work plans for d6c4d4f (2026-06-30) committed to git late on
2026-08-03 with all checkboxes still unchecked. Mark completed tasks done
and add a status note so they read as historical records, not open work.
- Refresh ROADMAP.md's "What's Next" section against current repo state.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Enables Rayon parallel compression for typical 4-chunk workloads (e.g.,
128×128 matrix with 32-row chunks). Rayon's dispatch overhead is ~2 µs,
worthwhile at ≥3 chunks with real compression work per chunk.
Previously the threshold was "> 4" which excluded 4-chunk datasets entirely
from parallel compression. Now "> 2" covers 3+ chunks.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
- Use Zstd level 3 instead of deflate(1) for embedding dataset compression.
Auto-shuffle (already the default since the TDT pre-filter commit) is now
the only shuffle needed — the explicit .with_shuffle() call was redundant.
- Benchmark: save_without_wal_single improves 67 → 61 µs (-9%).
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Implements WAL group commit optimizations (arXiv:2507.13062):
1. Serialize each WAL entry to a local Vec<u8> before writing, reducing
write() syscalls per entry from ~8 to 1.
2. Defer header entry_count updates to every GROUP_COMMIT_SIZE (8) entries
instead of per-entry, eliminating 3 lseek() + 1 write() per entry.
3. Fix read_entries() to read until EOF instead of looping entry_count
times — the header count is now a pre-allocation hint only. This is
strictly more robust: tolerates stale counts from deferred updates AND
truncated files from crashes mid-write.
Benchmark results:
- wal_flush_100_entries: -7.8% latency improvement (469 µs)
- save_with_wal_single: -1.7% (18.2 µs)
- save_without_wal_single: -2.3% (67 µs, full HDF5 write)
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Following arXiv:2506.18062 (TDT pre-filter) and matching h5py default
behavior: the shuffle filter is now automatically applied before any
compression codec (deflate, Zstd, LZ4, Pcodec) unless explicitly
disabled with .without_shuffle().
Benchmark results (f32 matrices, shuffle+codec vs unshuffled baseline):
- Zstd-3 at 512×512: 610 → 764 MiB/s (+25%)
- Deflate-6 at 128×128: 132 → 401 MiB/s (+204%)
- Deflate-6 at 512×512: 280 → 745 MiB/s (+166%)
Both codecs now reach parity at ~750 MiB/s for large matrices.
Changes:
- Add no_shuffle field to ChunkOptions (opt-out via .without_shuffle())
- Auto-add FILTER_SHUFFLE in build_pipeline() when compression is active
- Add DatasetBuilder.without_shuffle() method
- Update pipeline tests to reflect new 2-filter default
- Add chunk_options_pipeline_deflate_no_shuffle test
- Update BENCHMARKS.md with measured throughput improvements
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Implements Pcodec (filter ID 32023) via the `pco` 1.0.x crate as a new
optional compression codec. Pcodec achieves 30–94% better compression
ratio than Zstd for f32/f64 columnar data at 1–5 GiB/s decompression
speed, making it ideal for write-once/read-many embedding archives.
Write throughput at 512×512: 591 MiB/s (parity with Zstd-3 at 610 MiB/s).
For smaller chunks Zstd-3 remains faster due to Pcodec's fixed per-chunk
distributional analysis overhead.
- Add FILTER_PCODEC = 32023 constant to filter_pipeline.rs
- Add pcodec_compress/pcodec_decompress using pco::standalone API
- Wire into compress_chunk/decompress_chunk dispatch
- Add ChunkOptions.pcodec field and DatasetBuilder.with_pcodec() method
- Enable pcodec as highest-priority codec in build_pipeline()
- Add pco dep (optional, feature = "pcodec") to clawhdf5-format/clawhdf5
- Add write_2d_chunked_pcodec benchmark comparing pcodec vs zstd-3
- Document results in BENCHMARKS.md
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Add features = ["zstd"] to clawhdf5-bench dev-dependency so the
write_2d_chunked_zstd benchmark no longer panics with UnsupportedFilter(32015).
Update BENCHMARKS.md and README.md with measured results from the full
h5bench write suite (2026-06-30, post write-performance improvements):
- Zstd-3 hits 593 MiB/s at 512×512 vs deflate-6's 280 MiB/s (2.12×)
- Zstd-3 hits 330 MiB/s at 128×128 vs deflate-6's 132 MiB/s (2.51×)
- Sequential f64 batch write improved ~8-11% from owned-Vec IO path
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Four independent write-path improvements:
1. Cache compressed chunks between Pass 1 and Pass 2 (chunked_write.rs,
file_writer.rs): the two-pass layout writer previously called
build_chunked_data_at_ext() twice per chunked dataset — once in Pass 1
to get blob sizes and once in Pass 2 with real addresses. Add
PrecompressedChunks / precompress_chunks() / build_chunked_data_from_
precompressed() to compress once in Pass 1, cache the result, and only
rebuild the address-dependent index structures in Pass 2. Expected
~2× speedup for chunked+deflate writes (512×512 deflate: 3.33ms → ~1.7ms).
2. SIMD-vectorisable shuffle filter (filters.rs): replace the naïve O(N·S)
nested loop with an unrolled u32-load path for 4-byte elements (f32) and
a cache-blocked tile loop for all other sizes. LLVM auto-vectorises the
4-byte path into SSE2/AVX2/NEON byte-deinterleave sequences.
3. Zstd benchmark variant (h5bench_write.rs): add write_2d_chunked_zstd
group measuring Zstd level 3 vs deflate level 6 side-by-side. Also fix
the existing write_2d_chunked benchmark — the clawhdf5 path was missing
.with_deflate(6), making the comparison apples-to-oranges. Add arXiv-
backed doc recommendation on DatasetBuilder::with_zstd().
4. Zero-copy HNSW save (hnsw.rs, clawhdf5-io/lib.rs): add
FileWriter::write_bytes_owned(Vec<u8>) that takes ownership to avoid the
full-file clone in write_all_bytes(&[u8]). HNSW::save_to_hdf5 uses it.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Full head-to-head results from Criterion suite (100 samples each):
sequential read/write, chunked write + deflate, metadata ops, group
traversal. Includes interpretation section explaining the structural
reasons for each gap.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Critical fixes from whole-branch code review:
- libaec-sys: fix flag constants to match <libaec.h> exactly
(PREPROCESS=8, MSB=4, RESTRICTED=16; drop non-existent AEC_ALLOW_K13)
and add aec_buffer_encode FFI declaration
- filters_szip: fix cd index for bits_per_sample (cd[2] per H5Z_SZIP_PARM_BPP,
not cd[4]); fix option-mask mapping (NN=0x20, MSB unconditional); add two
real encode→decode roundtrip tests (no-NN and NN) that exercise libaec end-to-end
- file_writer: fix serialize_vds_mappings to delegate to data_layout_write
(eliminates the buggy duplicate that always emitted version=1 even for
external-file mappings); retains trailing Jenkins checksum
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
The previous aec_buffer_decode declaration used flat parameters which
don't match the actual libaec C API; this caused a SIGSEGV at runtime.
Replace with the correct aec_stream struct (mirroring <libaec.h>) and
update filters_szip.rs to populate and pass &mut AecStream.
Also add empty-input guard in szip_decode_impl and fallback library
path search in build.rs for distros that omit the .pc file.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Adds three Criterion benchmark suites mirroring the h5bench HPC I/O
benchmark workloads in pure Rust — no C libhdf5 required for the default
path, with an optional `libhdf5-compare` feature for side-by-side numbers.
- benches/h5bench_write.rs: write_1d_contiguous, write_2d_chunked,
write_f64_batch, write_multi_dataset, write_with_attrs
- benches/h5bench_read.rs: read_sequential, read_f64_sequential,
read_chunked_2d, read_from_disk, read_hyperslab
- benches/h5bench_meta.rs: metadata_attrs_write, metadata_attrs_read,
metadata_groups_create, metadata_groups_traverse, metadata_string_attrs
All benchmarks pass `cargo bench --bench <name> -- --test` and clippy
reports zero warnings. Run with `cargo bench -p clawhdf5-bench`.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Add `virtual_sources: Option<Vec<VdsMapping>>` field and `with_virtual_sources()` method to `DatasetBuilder`. In `FileWriter::finish()`, VDS datasets skip raw-data storage and instead serialize their source mappings into a global heap collection (version-1 same-file encoding) referenced by an HDF5 v4 layout-class-3 message. The two-pass address-computation loop handles VDS in both passes: pass 1 computes the fixed-size OH and pre-builds the heap blob; pass 2 places the blob at the correct file offset and rebuilds the OH with the real global heap address. Three new tests verify: (a) same-file two-source round-trip with mapping verification, (b) external-file source encoding, and (c) empty mapping list.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
clawhdf5-migrate:
- Real content validation: the post-migration check reads the written HDF5
back (new hdf5_reader) and compares actual content — chunk text, embeddings,
and every session/entity/relation field — to the source, not just row counts.
A representative sample of chunk rows is verified by default; --validate-full
checks every row. A count-preserving corruption no longer passes.
- Configurable schema: SQL is built from a SchemaConfig (table + ordered column
names, defaulting to the ZeroClaw layout) instead of hardcoded queries, with
--chunks-table / --sessions-table / --entities-table / --relations-table.
- Streaming count pass: --dry-run does a COUNT(*)-only pass per table instead
of loading every row.
- Incremental migration: --incremental reads the existing output, reads only
source chunks with id greater than the last migrated id, and appends them
(metadata groups refreshed from source) rather than re-migrating everything.
clawhdf5-format:
- read_as_f32 / read_as_f64 now decode IEEE-754 half-precision (2-byte) floats
via a no_std-safe bit conversion — needed to read float16-stored embeddings
back (e.g. for migrate's content validation), previously a TypeMismatch.
Tests: f16 read unit test; migrate tests for content-corruption detection,
custom table names, and incremental append; CLI smoke-tested end-to-end and the
dense/incremental output verified with h5py.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Dense attribute and dense link storage capped at a single fractal-heap direct
block (~64 KiB of heap data — a few thousand objects); beyond that the writer
produced an invalid oversized block. Lift the cap with a root indirect block.
When the serialized objects don't fit in one direct block, build a root
indirect block (FHIB) over multiple direct blocks sized by the doubling table
(start 512, width 4, doubling per row up to 64 KiB). Objects are packed
row-major across blocks, each block carries its logical block offset, and heap
IDs encode each object's heap offset (block offset + position). The FRHP points
root -> FHIB with the row count; unused slots in the current rows are undefined.
The fractal-heap builder is unified: FractalHeapBlock now carries the full heap
blob, and a shared write_frhp helper serializes the header for both the
single-block and multi-block paths. The single-block path is unchanged
(byte-identical), so existing dense attrs/links stay valid.
Validated end-to-end: a 2,500-attribute object and a 2,500-link group round-trip
through our reader and are read correctly by h5py. Objects still may not span a
block (no huge-object path).
Tests: facade round-trips for multi-block dense attrs and dense links, plus an
h5py-gated interop test (verified against the real h5py environment).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The fractal-heap reader split direct vs indirect block rows using the FRHP
"Starting # of Rows in Root Indirect Block" field (a constant, typically 1),
mislabeled as starting_row_of_indirect_blocks. For any heap whose data spans
more than one direct block — common in libhdf5 files with a large group or
many dense attributes — this treated direct blocks as indirect and walked into
garbage, failing with InvalidFractalHeapSignature.
Derive the split from the heap geometry instead: max_direct_rows =
log2(max_direct_block_size / starting_block_size) + 2. Rows below it hold
direct blocks; rows at/above hold child indirect blocks.
Validated against an h5py-written group with 400 dense attributes (root
indirect block, 4 rows, 13 direct blocks): all values now read correctly.
Regression fixture covers an 80-attribute multi-block heap.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
A group with more than 8 links (libhdf5's compact max_compact default) is now
written densely instead of as inline Link messages: the links live in a
single-direct-block fractal heap indexed by a v2 B-tree of type 5 (link-name
index), referenced from the group's LinkInfo message. This matches libhdf5's
compact->dense switchover and keeps large groups out of the object header.
- Extract the byte-identical fractal-heap builder shared by dense attributes
and dense links, parameterized by heap_id_length / max_heap_size. Attributes
keep 8 / 40; links use 7 / 32 to match libhdf5 (reverse-engineered: an
h5py-written dense group uses heap_id_length 7, max_heap_size 32, type-5
record = hash(4) + heap_id(7) = 11 bytes). This was the cause of an initial
"object overruns end of direct block" error from h5py.
- build_group_oh gained an optional dense LinkInfo (omitting inline Link
messages); the two-pass file assembly allocates each group's link blob after
its object header and rebuilds it with real target addresses in the final
pass (link-message size is address-independent, so layout is stable).
Validated end-to-end: our reader round-trips dense groups, h5py reads the
groups we write, and dense attributes remain byte-identical (still h5py-valid).
The agent's 9-dataset memory group now writes densely and round-trips. Single
direct block only (~a couple thousand links); indirect blocks remain a TODO.
Tests: facade round-trip (20-link dense group + compact sibling) and an
h5py-gated interop test confirming libhdf5 reads our dense groups.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The readers added this cycle parse untrusted bytes, so malformed/hostile
input must produce errors — never a panic, OOM, or unbounded recursion.
Audited each new surface and fixed the concrete vectors, each covered by an
adversarial regression test:
- Paged Fixed Array: `1 << max_nelmts_bits` shift overflow (u8 up to 255);
element-count bounded by file size; element/page offset multiplies checked.
- H5S selection decoder: ALL/NONE validate they have the 16 bytes they claim
to consume; hyperslab rank capped at 32 (H5S_MAX_RANK); iter_linear
coordinate/stride/product arithmetic uses checked ops.
- VDS mapping parser: drop pre-allocation from the untrusted `nused`;
bounds-check all selection slicing.
- scale-offset / N-Bit filters: `1 << minbits` overflow at minbits==64; N-Bit
`bit_offset + precision` overflow; N-Bit type-tree recursion depth capped to
stop a crafted nested tree from overflowing the stack; element counts bounded
by the chunk's expected decompressed size (threaded the previously-unused
chunk_size into both decoders) so a bogus count can't over-allocate.
- VDS assembly: a virtual dataset whose source is itself virtual (a cycle) now
errors instead of recursing into a stack overflow.
16 new adversarial tests; full format suite (482 lib) + agent + facade green;
clippy clean.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
clawhdf5-agent: fixed-length string datasets (memory text chunks, session
summaries, ids, tags, entity/relation names) were stored uncompressed behind
a stale "chunked compound not yet supported" comment. Chunked writes work for
fixed-size string/compound datatypes like any other, so write_string_dataset
now chunks + deflates once a dataset's payload reaches 4 KiB — large,
redundant NullPad content compresses well while tiny metadata stays
contiguous (no chunk-overhead bloat). The dead `compress` parameter is
removed in favor of this size heuristic.
clawhdf5-format: enabling string compression exposed a latent bug — the
per-file ChunkCache built its chunk index once and reused it for every
chunked dataset in the file, keyed only by chunk coordinate with no dataset
discrimination. With one chunked dataset per file this never surfaced; with
two of different rank (a 1-D compressed string array and the 2-D embeddings
matrix) the first dataset's rank-1 index was reused for the second, panicking
with an out-of-bounds chunk coordinate. The cache now binds to a dataset by
its chunk-index address and rebinds — dropping the index, chunk-index map,
layout, and decompressed slots — whenever the dataset being read changes,
while still caching repeated/sequential access to the same dataset.
Tests: facade regression reading a 1-D compressed string dataset and a 2-D
compressed f32 dataset through one shared File cache (verified to panic
without the fix); existing agent e2e tests (large text chunks, migration
round-trip) now pass with compression on.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The HDF5 library does not implement the scale-offset filter's floating-point
E-scale mode. When asked for it (cd_values[0] = 1) it stores the chunk raw
(no minbits/minval header) and sets the chunk filter mask to skip the filter,
so such datasets read back verbatim purely by honoring the per-chunk filter
mask — no E-scale decoder is required.
Add a fixture written via the HDF5 low-level API (exact-representable values)
and a test asserting it reads back verbatim, locking in the filter-mask path.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
VDS sources living in other files were previously unsupported because the
pure-byte read API has no filesystem. Add a resolver seam and wire a default.
clawhdf5-format:
- Add VdsSourceResolver (Fn(&str) -> Option<Vec<u8>>) and
read_raw_data_full_with_resolver. read_virtual_data uses the resolver to
fetch an external source file's bytes by its stored name, then reads the
named source dataset from those bytes and scatters as usual. A resolver
returning None leaves the region at fill (HDF5's missing-source behavior);
an external source with no resolver at all is a clean error. read_raw_data_full
is unchanged (delegates with no resolver).
clawhdf5:
- File now records the directory it was opened from and, for virtual layouts,
reads through a default resolver that loads sibling source files relative to
that directory. So File::open(virt).dataset(d).read_*() transparently
assembles cross-file VDS. In-memory files (from_bytes) have no directory, so
only same-file VDS resolves there.
Tests: format-layer external read with an injected resolver (and the
no-resolver error path), plus facade tests that drop both files in a temp dir
and read through File::open — covering successful resolution and the
missing-source-is-fill case.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Generalize Selection iteration from 1-D to arbitrary rank: iter_linear(dims)
enumerates a selection's row-major linear indices over a dataspace of the
given shape (ALL, NONE, regular hyperslabs, points), which is the order HDF5
uses to pair virtual and source selections.
read_virtual_data now passes the full virtual/source dimensions instead of a
single extent, so multi-dimensional block mappings scatter to the correct
non-contiguous linear positions. read_named_dataset_raw returns the source
dataset's dimensions. The rank-1 restriction is removed; only external-file
sources remain unsupported.
Tests: 2-D integration fixture (vds_2d_same_file.h5: two 2x2 sources placed
as non-contiguous blocks in a 4x4 virtual) plus N-D iter_linear unit tests
(block, strided, ALL, rank-mismatch). The 1-D path is unchanged.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
A virtual layout previously returned UnsupportedVersion. Implement reading
for the common 1-D, same-file case, reverse-engineered and validated against
HDF5 2.0.
- Rewrite parse_vds_mappings to the real global-heap block format
(version(1) · nused(length_size) · entries · checksum(4)), where each
entry is source-file(null) · source-dataset(null) · source-selection ·
virtual-selection. Block version 1 encodes a same-file source as a single
0x04 marker in place of the file name; version 0 stores an explicit file
name. The selections are H5S-serialized and self-describing in length, so
they are decoded to find entry boundaries. The previous parser used a
guessed layout that did not match real files.
- Extend Selection with decode_serialized() (H5S_select_serialize: ALL,
NONE, and version-3 regular hyperslabs) and iter_linear_1d().
- Add read_virtual_data: resolve the mapping block from the global heap,
read each same-file source dataset, and scatter its selected elements into
the virtual buffer; unmapped regions stay at the zero fill value.
External-file sources and N-D selections return a clean unsupported error.
Tests: real-file integration test (vds_same_file.h5: partial source slice +
fill gap), selection decoder unit tests built from the fixture bytes, and
same-file/external mapping-parser unit tests.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Omar Sobh
osobh