Listing a group read every child's object header and stopped at the
first that was not fetched yet, and so did the traversals of the group's
index (v1 B-tree and symbol table nodes, the local heap's names, v2
B-tree nodes and fractal heap objects). Over openUrl's restartable
reader each block cost its own pass and round trip: 184 serial requests
to list 3000 datasets at 1 MiB blocks, 536 at 64 KiB.
- core::Reader::list reads every child's header before returning the
first error (the same error, in listing order, Group::groups/datasets
return), classifying them as those do.
- clawhdf5-format: after the first sibling that fails, the B-tree v1
and v2 collectors, the symbol table node loop and the dense-link loop
go on reading (not using) the remaining siblings, then return that
first error: results and errors are unchanged, only failing
traversals read more, and in memory that is free (storage::touch).
A v1 group's local heap segment (names) is read at once, up to 1 MiB.
- LazyStorage no longer fills a one-block hole that is already cached
(it was fetched again: 215 MB fetched from a 198 MB file).
Measured with tests/lazy.rs listing_cost_of_a_given_file on the
reviewer's file (h5py, 3000 datasets of 64 KiB, 198 MB), list('/'):
libver earliest, 1 MiB blocks: 185 passes/184 requests -> 6/73
libver earliest, 64 KiB: 537/536 -> 8/531 (6 in flight)
libver latest, 1 MiB: 189/188 -> 9/98
libver latest, 64 KiB: 453/452 -> 11/452
Bytes fetched are unchanged (the headers are spread through the file).
New test listing_a_large_group_takes_a_few_passes (512-byte blocks):
FileBuilder 600 children 102 -> 5 passes; h5py earliest/latest 2000
children 8 and 11 passes. Conformance 600 of 697 (baseline 600);
check-32bit-casts, check-nostd and h5rs-fuzz over the CVE corpus clean.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Every `*_in` core and the read helpers take `file: &S` with
`S: Storage + ?Sized` instead of `&dyn Storage`, and the `&[u8]`
wrappers pass the slice itself, so they compile to a `[u8]` instance:
`as_contiguous()` inlines to `Some(self)` and each structure read is the
slice code's bounds check again, with no indirect call. `&dyn Storage`
still works (`S = dyn Storage`); there is one parser implementation.
Also, so the structure reads cost no more than the slice checks did:
- ObjectHeader::parse_in reads the prefix once (signature included)
instead of the signature and then the prefix: two reads for a
one-chunk header instead of three on a range backend;
- the symbol-table node and group B-tree (v1) loops walk their entries
with chunks_exact over the bytes read, and the node's redundant second
bounds check is gone (the entries' read is the check, same error);
- a version-1 header's message list is sized from its (capped) count.
Same results and errors; the unit and equivalence tests are unchanged.
New Criterion bench `clawhdf5/benches/local_metadata_bench.rs` over a
400-group version-1 file written by h5py (new fixture
`v1_groups_400.h5`): ObjectHeader::parse, symbol-table nodes, the group
B-tree walk and a facade listing, using only APIs that exist at f2ff2c4
so it builds there for an A/B.
Provisional A/B against f2ff2c4 (busy machine, not for docs): both
builds linked into one binary and timed in alternation, 200 rounds;
median ratio new/old: facade listing -0.5% to -3.5% (was +14%),
ObjectHeader::parse +1% to +2% (was +25%), symbol-table nodes -18%,
group B-tree walk -18%, local-heap names and resolve_group_children
within +-1.5%. An old-vs-old-copy run shows +-2% from code layout alone.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Every `u64 as usize` cast in clawhdf5-format (115 on wasm32) now goes
through addr::to_usize for values read from the file — addresses, lengths,
counts, dimensions: FormatError::Overflow where the value does not fit
instead of wrapping onto another part of the file on a 32-bit target — or
addr::saturating_usize for counts bounded by something in memory (codec
progress counters, writer sizes), which fail a bounds check or allocation
rather than wrap. A chunk whose offset does not fit lies outside the
dataset and is skipped; partial reads treat such an offset as out of the
buffers. On 64-bit targets nothing changes.
scripts/check-32bit-casts.sh (run by ci-test.sh) lints the wasm32 build
with clippy's cast_possible_truncation and fails on any u64 -> usize
finding; before this commit it listed 115.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
BTreeV1Node::parse_in reads a node's header and then its keys and
children, two bounded reads; collect_symbol_table_nodes_in walks the tree
over any Storage. The &[u8] functions are wrappers. New test: nodes with
siblings and 4- and 8-byte offsets cut at every length, and a two-level
tree with truncated leaves, give identical results through a read_at-only
CountingStorage (six reads for the three nodes).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
- Fix version skew: clawhdf5-py (pyproject.toml 1.93.0 -> 2.1.0) and
packages/clawhdf5-node (package.json 2.0.0 -> 2.1.0) were both behind
the actual crate version.
- Correct stale ROADMAP.md claims: the TypeScript bridge already has a
complete napi-rs package (not "no package.json"); CI/CD is now wired
up via .gitea/workflows/ci.yml.
- Fix CLAUDE.md: clawhdf5-gpu uses wgpu with hand-written WGSL compute
shaders, not CubeCL.
- chunked_read.rs: drop 12 unnecessary chunk_dimensions[..rank].to_vec()
allocations — all three callees already accept &[u32].
- btree_v1.rs: add an overflow-safe ensure_len(data, offset, needed)
helper (checked_add) and use it at the two plain-arithmetic bounds
guards, closing a usize-overflow edge case reachable from a crafted
near-usize::MAX B-tree offset. Add a regression test.
- Clarify that the integrity hashes in clawhdf5-agent/provenance.rs
(FNV-1a) and clawhdf5-format/provenance.rs (SHA-256) are unkeyed and
only detect accidental corruption, not tampering — doc-only change.
- README.md: document that the mpi-io feature's read/write paths are
root-read+broadcast / gather-to-rank-0, not true collective I/O.
