Merge branch 'perf/wasm-listing-passes' into feat/listing-header-last-files
# Conflicts: # CHANGELOG.md
This commit is contained in:
@@ -607,6 +607,16 @@ fast path within benchmark noise.
|
||||
missing blocks: listing 3000 datasets went from 185 passes to 6.
|
||||
The 32-bit risk below is covered by a Node test that reads data at
|
||||
3 GiB from a mock server and is refused a 4 GiB file.
|
||||
- Fewer round trips (2026-09-27, later): the walks descend into every
|
||||
child after a failure (not only read the siblings), and parsers
|
||||
call `Storage::hint` for what they read next (node bodies, object
|
||||
header chunks, a dense group's heap blocks, a listing's child
|
||||
headers); `LazyStorage` fetches hinted blocks only along with a
|
||||
pass's real misses and within `maxFetch`, so hints never add a
|
||||
round trip nor change a result. A v1 group's name is looked up down
|
||||
its B-tree (`H5G__stab_lookup`), not by listing it. 3000 datasets
|
||||
list in 4 passes (earliest) and 5 (latest) at 1 MiB blocks, and
|
||||
opening one of them costs 5 requests, not 74 (CHANGELOG).
|
||||
|
||||
**M5 — SWMR and growth (later, separate design).** `Storage::len()` may grow;
|
||||
add `File::refresh()` that re-reads the superblock/EOF and invalidates cached
|
||||
|
||||
+23
-4
@@ -1104,10 +1104,29 @@ cache, but:
|
||||
header, and every node of a level of the group's index, in one pass
|
||||
(since 2026-09-27; it was one round trip per header block): 3000
|
||||
datasets of an h5py file took 6 passes at 1 MiB blocks, 9 for a
|
||||
`libver="latest"` file (dense links). Each pass re-parses what the
|
||||
call reads (CPU, not network). With headers spread through the file
|
||||
(h5py writes each next to its data) a listing still fetches most of
|
||||
the file at 1 MiB blocks; a smaller `blockSize` fetches less.
|
||||
`libver="latest"` file (dense links). **Since 2026-09-27 (later):**
|
||||
4 and 5 passes (5 and 6 at 64 KiB, from 8 and 11): the index walks go
|
||||
on past a missing node, and parsers hint what they read next
|
||||
(`Storage::hint`: node bodies, the heap's blocks, each child's
|
||||
header), which the lazy reader fetches with a pass's misses. That is
|
||||
the depth of the chain (index levels, then symbol table nodes or
|
||||
heap objects, then headers) plus the pass that finishes; it cannot
|
||||
go lower without reading structures before their addresses are
|
||||
known. Opening one dataset of a v1 group looks its name up down the
|
||||
group's B-tree (it read every entry: 74 requests, 193 MB at 1 MiB
|
||||
blocks for one 64 KiB dataset of the 3000; now 5 requests, 5 MB).
|
||||
Each pass re-parses what the call reads (CPU, not network). With
|
||||
headers spread through the file (h5py writes each next to its data)
|
||||
a listing still fetches most of the file at 1 MiB blocks (192 of
|
||||
198 MB; 35 MB in 530 requests at 64 KiB); a smaller `blockSize`
|
||||
fetches less. Merging nearby requests does not help such a file: the
|
||||
blocks a listing needs are five or six apart at 64 KiB, so fewer requests would
|
||||
mean fetching most of the file. Listing it a second time is free at
|
||||
64 KiB blocks, but at 1 MiB its metadata blocks (192 MB) exceed the
|
||||
64 MiB `cacheSize`, so they are fetched again (the earliest file: 4
|
||||
passes, 50 requests); a larger `cacheSize` keeps them. A file's paged
|
||||
aggregation (metadata in pages) is not used to fetch its metadata in
|
||||
one request.
|
||||
- **Memory:** a call keeps every block it reads until it finishes (the
|
||||
cache budget applies between calls). It may fetch at most `maxFetch`
|
||||
bytes (512 MiB by default, at most 1 GiB), and a single read longer
|
||||
|
||||
Reference in New Issue
Block a user