docs: fewer round trips for remote files in the browser, counted
CHANGELOG (Unreleased): the v1 B-tree lookup, `Storage::hint`, the walks that go on past a missing node, and the counts before and after on an h5py file like the reviewer's (3000 datasets, 198 MB, earliest and latest libver, 1 MiB and 64 KiB blocks), the corpus read lazily at 512 B and 64 KiB blocks, and the Node/Chromium suite. known-issues (browser limits, round trips): the new counts, why the passes cannot go lower (the chain of addresses), why merging nearby requests does not help such a file, and that a second listing refetches at 1 MiB blocks when the file's metadata blocks exceed `cacheSize`. range-reads.md M4 status and the viewer README follow. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
+23
-4
@@ -1051,10 +1051,29 @@ cache, but:
|
||||
header, and every node of a level of the group's index, in one pass
|
||||
(since 2026-09-27; it was one round trip per header block): 3000
|
||||
datasets of an h5py file took 6 passes at 1 MiB blocks, 9 for a
|
||||
`libver="latest"` file (dense links). Each pass re-parses what the
|
||||
call reads (CPU, not network). With headers spread through the file
|
||||
(h5py writes each next to its data) a listing still fetches most of
|
||||
the file at 1 MiB blocks; a smaller `blockSize` fetches less.
|
||||
`libver="latest"` file (dense links). **Since 2026-09-27 (later):**
|
||||
4 and 5 passes (5 and 6 at 64 KiB, from 8 and 11): the index walks go
|
||||
on past a missing node, and parsers hint what they read next
|
||||
(`Storage::hint`: node bodies, the heap's blocks, each child's
|
||||
header), which the lazy reader fetches with a pass's misses. That is
|
||||
the depth of the chain (index levels, then symbol table nodes or
|
||||
heap objects, then headers) plus the pass that finishes; it cannot
|
||||
go lower without reading structures before their addresses are
|
||||
known. Opening one dataset of a v1 group looks its name up down the
|
||||
group's B-tree (it read every entry: 74 requests, 193 MB at 1 MiB
|
||||
blocks for one 64 KiB dataset of the 3000; now 5 requests, 5 MB).
|
||||
Each pass re-parses what the call reads (CPU, not network). With
|
||||
headers spread through the file (h5py writes each next to its data)
|
||||
a listing still fetches most of the file at 1 MiB blocks (192 of
|
||||
198 MB; 35 MB in 530 requests at 64 KiB); a smaller `blockSize`
|
||||
fetches less. Merging nearby requests does not help such a file: the
|
||||
blocks a listing needs are five or six apart at 64 KiB, so fewer requests would
|
||||
mean fetching most of the file. Listing it a second time is free at
|
||||
64 KiB blocks, but at 1 MiB its metadata blocks (192 MB) exceed the
|
||||
64 MiB `cacheSize`, so they are fetched again (the earliest file: 4
|
||||
passes, 50 requests); a larger `cacheSize` keeps them. A file's paged
|
||||
aggregation (metadata in pages) is not used to fetch its metadata in
|
||||
one request.
|
||||
- **Memory:** a call keeps every block it reads until it finishes (the
|
||||
cache budget applies between calls). It may fetch at most `maxFetch`
|
||||
bytes (512 MiB by default, at most 1 GiB), and a single read longer
|
||||
|
||||
Reference in New Issue
Block a user