Merge branch 'perf/wasm-listing-passes' into feat/listing-header-last-files

# Conflicts:
#	CHANGELOG.md
This commit is contained in:
osobh
2026-09-27 23:16:27 -05:00
16 changed files with 1039 additions and 134 deletions
+23 -4
View File
@@ -1104,10 +1104,29 @@ cache, but:
header, and every node of a level of the group's index, in one pass
(since 2026-09-27; it was one round trip per header block): 3000
datasets of an h5py file took 6 passes at 1 MiB blocks, 9 for a
`libver="latest"` file (dense links). Each pass re-parses what the
call reads (CPU, not network). With headers spread through the file
(h5py writes each next to its data) a listing still fetches most of
the file at 1 MiB blocks; a smaller `blockSize` fetches less.
`libver="latest"` file (dense links). **Since 2026-09-27 (later):**
4 and 5 passes (5 and 6 at 64 KiB, from 8 and 11): the index walks go
on past a missing node, and parsers hint what they read next
(`Storage::hint`: node bodies, the heap's blocks, each child's
header), which the lazy reader fetches with a pass's misses. That is
the depth of the chain (index levels, then symbol table nodes or
heap objects, then headers) plus the pass that finishes; it cannot
go lower without reading structures before their addresses are
known. Opening one dataset of a v1 group looks its name up down the
group's B-tree (it read every entry: 74 requests, 193 MB at 1 MiB
blocks for one 64 KiB dataset of the 3000; now 5 requests, 5 MB).
Each pass re-parses what the call reads (CPU, not network). With
headers spread through the file (h5py writes each next to its data)
a listing still fetches most of the file at 1 MiB blocks (192 of
198 MB; 35 MB in 530 requests at 64 KiB); a smaller `blockSize`
fetches less. Merging nearby requests does not help such a file: the
blocks a listing needs are five or six apart at 64 KiB, so fewer requests would
mean fetching most of the file. Listing it a second time is free at
64 KiB blocks, but at 1 MiB its metadata blocks (192 MB) exceed the
64 MiB `cacheSize`, so they are fetched again (the earliest file: 4
passes, 50 requests); a larger `cacheSize` keeps them. A file's paged
aggregation (metadata in pages) is not used to fetch its metadata in
one request.
- **Memory:** a call keeps every block it reads until it finishes (the
cache budget applies between calls). It may fetch at most `maxFetch`
bytes (512 MiB by default, at most 1 GiB), and a single read longer