Listing a large group over openUrl still took 6-11 passes (network round
trips) for the reviewer's 3000-dataset h5py file: each pass only found
the structures the walk reached before its first miss.
- The v1 and v2 B-tree collectors descend into every child of a node
after one fails (they only read the siblings before, so a sibling's
subtree came a pass later), then return the first error: results and
errors unchanged. The v2 walk stops once its record budget is spent,
so a shared-subtree tree still cannot multiply the work.
- Hints (`Storage::hint`, a no-op for every backend but the lazy one):
a group B-tree node's and a symbol table node's body (read once their
header gives a length, a round trip later when the body is in the
next block), an object header's first chunk and its continuation
chunks, the symbol table nodes a B-tree leaf names, a dense group's
name index header and the heap's root block (both read right after
the heap header). A listing also hints every child's object header as
its entry is read, even after a failure, and every direct block of a
dense group's heap (reading the indirect blocks, at most 4096 entries
and 4 levels deep); a lookup does not.
- The fractal heap's indirect-block layout (entry sizes, where the first
n entries end) is one helper used by the object reads and the hints.
Measured with tests/lazy.rs listing_cost_of_a_given_file on an h5py file
like the reviewer's (3000 datasets of 64 KiB, 198 MB), list('/'),
passes/requests/bytes, before -> after:
earliest, 1 MiB: 6/73/192.5 MB -> 4/68/192.5 MB
earliest, 64 KiB: 8/531/35.2 MB -> 5/530/35.3 MB
latest, 1 MiB: 9/98/196.5 MB -> 5/86/196.5 MB
latest, 64 KiB: 11/452/29.6 MB -> 6/454/30.5 MB
listing_a_large_group_takes_a_few_passes (512-byte blocks), budgets
tightened to the new counts: FileBuilder 600 children 5 -> 4 passes,
h5py 2000 children earliest 8 -> 5, latest 11 -> 6.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Every `*_in` core and the read helpers take `file: &S` with
`S: Storage + ?Sized` instead of `&dyn Storage`, and the `&[u8]`
wrappers pass the slice itself, so they compile to a `[u8]` instance:
`as_contiguous()` inlines to `Some(self)` and each structure read is the
slice code's bounds check again, with no indirect call. `&dyn Storage`
still works (`S = dyn Storage`); there is one parser implementation.
Also, so the structure reads cost no more than the slice checks did:
- ObjectHeader::parse_in reads the prefix once (signature included)
instead of the signature and then the prefix: two reads for a
one-chunk header instead of three on a range backend;
- the symbol-table node and group B-tree (v1) loops walk their entries
with chunks_exact over the bytes read, and the node's redundant second
bounds check is gone (the entries' read is the check, same error);
- a version-1 header's message list is sized from its (capped) count.
Same results and errors; the unit and equivalence tests are unchanged.
New Criterion bench `clawhdf5/benches/local_metadata_bench.rs` over a
400-group version-1 file written by h5py (new fixture
`v1_groups_400.h5`): ObjectHeader::parse, symbol-table nodes, the group
B-tree walk and a facade listing, using only APIs that exist at f2ff2c4
so it builds there for an A/B.
Provisional A/B against f2ff2c4 (busy machine, not for docs): both
builds linked into one binary and timed in alternation, 200 rounds;
median ratio new/old: facade listing -0.5% to -3.5% (was +14%),
ObjectHeader::parse +1% to +2% (was +25%), symbol-table nodes -18%,
group B-tree walk -18%, local-heap names and resolve_group_children
within +-1.5%. An old-vs-old-copy run shows +-2% from code layout alone.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
SymbolTableNode::parse_in reads the node's 8-byte header, checks the
entries against the end of the file as before, and reads them in one
bounded read. parse is a wrapper. New test: nodes with 4- and 8-byte
offsets, valid and with a bad version, at two offsets and cut at every
length, parse identically through a read_at-only CountingStorage.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Formatting only. cargo fmt --check was already failing on main (accel SIMD
kernels, agent, format, migrate, bench); CI now enforces it.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
SymbolTableNode::parse used raw offset+8 arithmetic that can overflow
on a crafted v1-group B-tree leaf with a near-u64::MAX SNOD child
pointer (group_v1.rs passes such offsets through unchecked). Switch to
checked_add, matching read_offset in the same file. Also harden the
entries_start + num_symbols*entry_size computation with checked_add
for consistency, even though num_symbols being u16 already bounds
that multiply. Add regression tests.
INT-02
