format: object headers without per-header allocations for their chunk list

Reading continuation chunks from a queue (7e5e920, a69c5be) allocated a
queue Vec and a BTreeSet of chunk starts for every header, and inlined
the per-chunk message loop into the generic parser: ObjectHeader::parse
over 401 version-1 headers went from 24.8 to 45.7 us.

ChunkSpans now keeps the first 8 chunks in an inline array (cycle check
by scan) and is also the read queue; only a header of more chunks
allocates (a boxed spill list and start set). The message loop of one
version-1 chunk is its own non-generic function. Same checks as before:
any number of chunks up to 65,536, cycles refused, chunks bounded by the
file size, one chunk buffer alive at a time, libhdf5 message order,
overlap allowed. The cycle test now also covers spilled chunk lists.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-26 20:30:16 -05:00
co-authored by Claude Opus 5.5
parent 011e0dbb96
commit 4313917b4d
2 changed files with 149 additions and 63 deletions
+8
View File
@@ -295,6 +295,14 @@
chunks nested in each other made storage with owned buffers read and
hold the square of the file's size), or more than 65 536 chunks are
refused, so a header's chunks read at most the file's size.
The first version allocated a queue and a set of chunk starts for every
header and made `ObjectHeader::parse` of 401 small headers 1.8x slower
(`local_metadata_bench`: 45.7 vs 24.8 µs on `main` `8f59b2e`). The first
8 chunks now live in an inline array (cycle check by scan; only a longer
header allocates), and the per-chunk message loop is its own function
instead of being inlined into the generic parser. Provisional (tank load
3–4; Criterion, separate binaries, 2 alternating rounds): 24.96–25.12 vs
24.85–25.15 µs on `main`.
- Tests: `crates/clawhdf5-tools/tests/edit_coverage_interop.rs` (h5py
`earliest`/`v110`/`latest` and clawhdf5-written files; structure
comparisons with libhdf5 for version-2 B-trees, shrink on every index,