diff --git a/docs/design/range-reads.md b/docs/design/range-reads.md new file mode 100644 index 0000000..2e82a6c --- /dev/null +++ b/docs/design/range-reads.md @@ -0,0 +1,499 @@ +# Design: range reads (reading HDF5 without holding the whole file) + +Status: proposal, 2026-09-26. No library code has changed; this document is +the plan for Phase 3's largest architectural change. Every count below was +taken on `tank` on 2026-09-26 at commit `de2a53f`, with the commands given +next to it. No timing numbers appear here on purpose: the machine was shared +with other build jobs when this was written. + +## The problem + +Every reader in clawhdf5 parses the file through one `&[u8]` that covers the +whole file: + +- `clawhdf5_io::HDF5Read::as_bytes(&self) -> &[u8]` is the only read method of + the I/O trait (`crates/clawhdf5-io/src/lib.rs`). `FileReader` reads the + whole file into a `Vec`, `MmapReader` maps it. +- The facade's `File` keeps a `Backing` (`Vec` or mmap) and hands + `FileData::as_bytes()` — the file from the superblock to the recorded end of + file — to every `clawhdf5-format` call (`crates/clawhdf5/src/reader.rs`). +- The format crate's parsers take `file_data: &[u8]` plus a `usize` address and + slice into it. +- `LazyFile` (facade) parses metadata lazily, but still over + `HDF5Read::as_bytes`. `AsyncHDF5Read` (`clawhdf5-io`, `async` feature) + already has `read_at(offset, len)`, but `AsyncHDF5File::open` calls + `read_all()` and parses the result. +- The browser reader (`clawhdf5-wasm`) is `open(bytes: Vec)`: the page + must download the whole file before it can list a group. + +That rules out four things users of other HDF5 readers have: + +1. **Remote files by range request** — libhdf5's `ros3` driver, h5py over + `fsspec`/`s3fs`, pyfive over `fsspec`. Today a 30 GB file on S3 has to be + downloaded before its tree can be listed. +2. **Lazy loading in the browser** — h5wasm can back a file with HTTP range + requests; our wasm reader holds the whole file. +3. **SWMR readers of a growing file** — the slice is fixed at open; a reader + cannot see the file grow (and an mmap of a growing file is fragile). +4. **Files larger than the address space on 32-bit targets** — including + `wasm32`, where `usize` is 32 bits and a file above 4 GiB cannot even be put + in a slice; browsers cap `ArrayBuffer`s lower than that. + +## 1. Inventory + +### 1.1 Functions that take the whole file + +`python3 docs/design/tools/inventory.py` scans non-test source (code before a +file's `#[cfg(test)] mod`, no `tests/`) for functions with a +`file_data: &[u8]` parameter (the repository's convention), and — as a +heuristic — functions with a `data`/`buf`/`file`/`bytes` slice *and* an +integer address/offset parameter. `--list` prints every hit. + +- **103 functions take `file_data: &[u8]`** (73 of them `pub`), in 25 files. +- The heuristic adds 27, of which only **6 really take the whole file under + another name**: `ObjectHeader::parse` and its internal `parse_v1`, + `parse_v1_chunk`, `parse_v2`, `parse_v2_continuation` (`data: &[u8]`), and + `Superblock::parse`. The other 21 are helpers (`ensure_len`, + `read_length`, `read_uint`, …) that bound-check whatever slice they are + given, or writer-side `write_at`s in `clawhdf5-io`. + +So **109 functions** have to change signature. Grouped by what they read: + +| Read pattern | Modules (whole-file functions) | Count | +|---|---|---:| +| **Small metadata reads at addresses** — a structure whose size is usually only known after its prefix is read | `group_v2` 9, `shared_message` 9, `fractal_heap` 7, `attribute` 6, `object_header` 5, `group_v1` 5, `extensible_array` 4, `btree_v2` 4, `btree_v1` 3, `fixed_array` 3, `local_heap` 3, `fill_value` 3, `superblock` 2, `data_layout` 1, `symbol_table` 1 | 65 | +| **Bulk raw data** — a contiguous extent, or a list of chunk extents known once the chunk index has been walked | `chunked_read` 12, `data_read` 9, `vds` 6, `parallel_read` 3, `partial_read` 1, `provenance` 1 | 32 | +| **Heap-resident data** — VL strings and sequences: many small reads into global-heap collections | `vl_data` 3, `global_heap` 2, facade `vlen.rs` 3, facade `types.rs` 3 | 11 | +| Bindings | `clawhdf5-py/src/convert.rs` | 1 | + +### 1.2 How they read + +`python3 docs/design/tools/inventory.py --patterns` (non-test code): + +| Pattern | Where | Count | +|---|---|---:| +| `file_data` passed on to another function (call sites) | format 262, facade 7, py 2 | 271 | +| `file_data[..]` slicing | format | 137 | +| of which open-ended `file_data[x..]` (slice to end of file) | format | 5 | +| `file_data.len()` (end-of-file bound checks) | format | 38 | +| address/offset `as usize` casts | format 120, facade 5, io 5, ann 3 | 133 | +| `.as_bytes()` on a file or reader, feeding format calls | facade 26, py 8, tools 1, wasm 1 | 36 | + +Three things in this table shape the design: + +- **Open-ended slices and `len()` checks** assume the whole file is present. + With a range reader "the rest of the file" is a request for gigabytes; each + of these 43 sites must become a bounded read. +- **The 133 `as usize` casts** are where a 64-bit HDF5 address is squeezed + into a pointer-sized index. On 32-bit targets they truncate or must fail. + A storage trait that takes `u64` offsets removes most of them. +- **The bytes escape through the public API.** `File::as_bytes()`, + `LazyFile::as_bytes()`, `MmapFile::as_bytes()` return the whole file, and 15 + facade methods return `&'f [u8]`/`&'f [T]` borrowed from it + (`Dataset::read_raw_ref`, `read_as_slice`, `read_*_zerocopy`). The Python + bindings (`clawhdf5-py/src/node.rs`, `attrs.rs`, …) call + `clawhdf5_format` directly with `file.as_bytes()`, and so do `h5rs` and the + wasm reader. These must keep working for local files (they are the + zero-copy fast path) and fail cleanly for remote ones. + +### 1.3 Pieces that already exist + +- `clawhdf5_format::metadata_cache::MetadataCache` — LRU keyed by file offset + with a byte budget (2 MiB default). Not currently in the read path. +- `clawhdf5_io::prefetch::PrefetchReader` — ring buffer of `(offset, len)` + entries over an `HDF5Read`, plus a sweep detector for chunk access patterns. +- `clawhdf5_io::async_read::AsyncHDF5Read::read_at` — the right shape of + trait, but async and only used to read everything. +- `clawhdf5_io::hsds` — a client for the HSDS REST service. Different + protocol (the server parses HDF5), not a range reader. + +## 2. Measurement: how chatty is a naive range reader? + +`docs/design/tools/range-trace/` loads a file into a page-aligned buffer, +opens it with `clawhdf5::File::from_bytes` (today's code, unchanged), +`mprotect`s the buffer and single-steps every load that faults (SIGSEGV logs +the exact address and unprotects the page; the x86 trap flag re-protects it +after one instruction). So every load instruction that touches the file is +recorded, with no change to the library. A page loaded more than 4096 times in +a phase is left readable and counted as wholly read. Phases: **open** +(superblock), **list** (walk every group; shape and dtype of every dataset — +what a tree view or `h5ls -r -v` needs), **read** (one dataset, whole). With +the dataset's chunk extents (from h5py, `libhdf5_reads.py --extents`) the read +phase is split into the chunk index and the raw data. + +Columns: *uncached requests* — a new request every time the access stream +leaves the neighbourhood (±64 B) of the current run, i.e. a reader that turns +each parse into a read with no cache; *ranges* — distinct byte ranges after +merging accesses closer than 64 B; *N KiB blocks* — distinct aligned blocks, +i.e. the requests a reader with a block cache of that size would make. For +comparison, `libhdf5_reads.py` opens the same file in h5py through a Python +file object (the `fileobj` driver — exactly how h5py + fsspec read remote +files) and counts libhdf5's `read` calls, default h5py settings. + +```bash +CARGO_TARGET_DIR=$PWD/target cargo build --release \ + --manifest-path docs/design/tools/range-trace/Cargo.toml +PY=/home/osobh/projects/clawhdf5/.venv/bin/python +$PY docs/design/tools/libhdf5_reads.py FILE DSET --extents > ext.txt +target/release/range-trace FILE DSET ext.txt +$PY docs/design/tools/libhdf5_reads.py FILE DSET +``` + +Files (from `conformance/.cache/corpus`), 2026-09-26 on tank: + +- **A** `xarray-data/imerghh_730.hdf5` — NASA IMERG, 7.7 MB, superblock v0, + 21 objects; read `/Grid/precipitation` (25 deflated chunks). +- **B** `NCAS-CMS_pyfive/tests/data/cmip_bad_eg.nc` — netCDF-4, 48 MB, + superblock v2, 9 objects; read `/tas` (780 deflated chunks, 43.7 MB stored). +- **C** `hdf5/tools/test/testfiles/h5stat_newgrat.h5` — 6.4 MB, superblock + v3, 35 001 groups in one dense (fractal heap + v2 B-tree) root group; read + `/DATASET_NAME` (a scalar with no storage). +- **B′** B repacked with paged aggregation: + `h5repack -S PAGE -G 65536` (55.5 MB). + +clawhdf5 (range-trace): + +| File | Phase | Uncached requests | Ranges | Bytes in ranges | 4 KiB blocks | 64 KiB blocks | 1 MiB blocks | +|---|---|---:|---:|---:|---:|---:|---:| +| A | open + list | 410 | 5 | 25 285 | 8 | 3 | 2 | +| A | read: chunk index | 33 | 8 | 4 608 | 7 | 3 | 2 | +| A | read: raw data | (25 chunks) | 2 | 1 496 533 | 368 | 25 | 3 | +| A | all metadata | 443 | 6 | 26 549 | 10 | 3 | 2 | +| B | open + list | 143 | 5 | 17 951 | 7 | 1 | 1 | +| B | read: chunk index | 46 | 19 | 49 583 | 28 | 15 | 14 | +| B | read: raw data | (780 chunks) | 15 | 43 656 557 | 10 671 | 668 | 43 | +| B | all metadata | 189 | 20 | 57 023 | 30 | 15 | 14 | +| B′ | all metadata | 413 | 24 | 71 083 | 25 | 3 | 2 | +| B′ | read: raw data | (780 chunks) | 780 | 43 613 717 | 10 920 | 780 | 49 | +| C | open + list | 455 779 | 3 | 6 357 627 | 1 553 | 98 | 7 | +| C | read: resolve `/DATASET_NAME` | 38 525 | 936 | 1 137 116 | 948 | 98 | 7 | + +libhdf5 through h5py's `fileobj` driver (read calls, bytes): + +| File | open | list | read | total calls | total bytes | +|---|---:|---:|---:|---:|---:| +| A | 3 | 93 | 25 | 121 | 1 563 559 | +| B | 2 | 78 | 780 | 860 | 43 769 952 | +| C | 4 | 36 105 | 1 | 36 110 | 19 725 823 | + +What this says: + +1. **A reader with no cache is hopeless.** clawhdf5's parsers revisit the same + structures many times (A: 26 000 loads to list 21 objects: + `Group::dataset(name)` and `Group::group(name)` re-read the group's whole + link list for every lookup). Mapped + one-to-one onto requests that is 410 round trips to list 21 objects, and + 455 779 to list C. **A cache is not an optimisation, it is the design.** +2. **With a block cache, metadata is cheap on typical files.** Metadata of A + and B touches 5–20 ranges and 2–15 blocks of 1 MiB; a 1 MiB-block cache + lists A in 2 requests and B in 1, where libhdf5 issues 96 and 80 calls + (fsspec's block cache absorbs those for h5py). +3. **"Prefetch the metadata region" alone does not work.** In B the chunk + index (v1 B-tree nodes) is interleaved with the raw data: the metadata + touches 14 distinct 1 MiB blocks spread over 48 MB. Only 4 of the 611 + corpus files h5py can open use paged aggregation (all four are libhdf5 + test files; 502 have a v0 superblock). Repacked as B′, the same metadata + fits in 3 blocks of 64 KiB — paging helps a lot when present, but a reader + cannot count on it. +4. **Raw data is naturally a batch.** Once the chunk index is walked, all + chunk extents are known: B's 780 chunks are 15 byte ranges after merging + neighbours, so a coalescing `get_ranges` call reads the dataset in a + handful of parallel requests. libhdf5 issues one read per chunk (780). +5. **An existing inefficiency becomes a blocker.** Resolving `/DATASET_NAME` + in C reads 1.1 MB (936 ranges) because + `group_v2::resolve_path_following_links` enumerates every link of the + group (`resolve_group_entries`) and compares names, instead of hashing + the name and descending the v2 B-tree name index. On an mmap this is + merely slow; over HTTP it is 98 requests of 64 KiB for one lookup. + libhdf5 reads one 512-byte block. Listing C is inherently whole-file (35 001 + object headers spread over the file), and libhdf5 reads 19.7 MB — three + times the file — to do it. + +Caveats: loads are logged with their first byte and a nominal width of 8 B, +so byte totals are approximate (±64 B per range); "uncached requests" is a +model, not a measured reader. These are counts, not timings. + +## 3. Options + +### (a) A storage trait threaded through the format crate + +```rust +// clawhdf5-format (no_std + alloc) +pub trait Storage { + /// Bytes [offset, offset + len). Short only at end of file. + fn read_at(&self, offset: u64, len: usize) -> Result, FormatError>; + /// Current length; may grow between calls (SWMR). + fn len(&self) -> u64; + /// Batch read; backends coalesce and parallelise. Default: loop. + fn read_ranges(&self, ranges: &[Range]) -> Result>, FormatError> { .. } + /// The whole file as one slice, when the backend has it (Vec, mmap). + fn as_contiguous(&self) -> Option<&[u8]> { None } +} +impl Storage for [u8] { /* Cow::Borrowed, as_contiguous = Some(self) */ } +``` + +Every `file_data: &[u8]` becomes `file: &dyn Storage` (or `&S` generic), and +every `file_data[a..b]` becomes `file.read_at(a, b - a)?`. + +- **For:** explicit, sound, errors are values (a network error is an `Err`, + not a panic); works for any backend; `u64` offsets fix 32-bit; `len()` can + grow, which is what SWMR needs; `read_ranges` gives raw data its natural + batch shape; `impl Storage for [u8]` lets the migration go one module at a + time with `&[u8]` callers unchanged. +- **Against:** touches all 109 functions and ~400 slice/len/call sites + (§1.2); every parse that learns a structure's size from its prefix needs two + reads (prefix, then body) — cheap against a cache, but it must be written + that way. `Cow::Owned` results cost a copy per read for remote backends. +- **Sync vs async:** the format crate is `no_std` and synchronous and should + stay so (parsing is CPU work; an async parser would colour 109 functions and + their callers). Remote backends are async. Two bridges, both needed: + - native: a backend that blocks on its own runtime (`object_store` + + a private tokio runtime), used from ordinary threads; + - wasm (no blocking on the main thread): a **restartable** mode where the + cache returns `FormatError::NeedBytes { offset, len }` on a miss; an async + driver fetches the block and re-runs the (pure, idempotent) operation. + parquet-rs's async reader uses the same "fetch, then parse synchronously" + split. The measurement bounds the retries: one per missed block, i.e. 1–15 + for A and B's metadata at 1 MiB blocks. +- **`&dyn` vs generic:** generics monomorphise 109 functions per backend + (binary size matters for wasm); `&dyn Storage` costs one indirect call per + structure read, negligible next to parsing. Hot raw-data loops keep their + speed through `as_contiguous()`. + +### (b) A page-cache "virtual slice" + +A type that implements `Index, Output = [u8]>` and faults pages +in on demand from interior-mutable storage, so parsers keep their slicing +syntax. + +- `Index::index` returns `&[u8]` borrowed from `&self`. A cache that evicts + pages invalidates references that are still alive — unsound — so pages can + never be evicted while the object lives (unbounded memory, which is the + problem we are solving). +- A range that crosses a page boundary has to be copied into a contiguous + buffer that also must outlive the borrow: an append-only arena that only + grows. +- `Index` cannot fail: a network error or truncated response becomes a + panic, which breaks "never return wrong data; unsupported is a clean + error". +- Open-ended slices (`file_data[x..]`) and `file_data.len()` still mean "the + whole rest of the file". +- It hides the cost model: a `for` loop over `file_data[i]` becomes thousands + of cache lookups, and nothing in the signature says the call can block. + +Rejected. It saves the signature churn of (a) but moves every failure into +panics and unsoundness. + +### (c) Keep `&[u8]` for metadata by prefetching it; range-read raw data only + +Fetch "the metadata" up front into a buffer, parse it with today's code, and +range-read only chunk data. + +- HDF5 has no pointer to a metadata region. libhdf5's paged aggregation + (`H5Pset_file_space_strategy(H5F_FSPACE_STRATEGY_PAGE)`, + `h5repack -S PAGE -G `) keeps metadata and raw data in separate pages, + and "setting an appropriate page size can have all internal file metadata + in just one page" ([Cloud-Optimized HDF/NetCDF guide][cog]) — but only + when the writer chose it: 4 of 611 corpus files (§2). +- Without paging, metadata is scattered: in B the chunk index is spread over + 14 MiB-blocks of a 48 MB file. Prefetching the first N MiB (what earthaccess' + block cache effectively does for "shared metadata at the file beginning" + ([earthaccess][ea])) is a good *heuristic*, not a correctness basis. +- A sparse buffer still presents as a `&[u8]` of the whole file's length, so + every metadata read outside the prefetched part must be caught — back to (b). + +Useful as a **policy** on top of (a) (prefetch the first block, use the page +size of paged files as the block size), not as the architecture. + +### (d) mmap a userfaultfd- or FUSE-backed file + +Keep `&[u8]` everywhere and let the kernel fault in remote pages +(userfaultfd handler, or a FUSE file system doing range GETs). + +- Linux-only (userfaultfd) or needs a FUSE mount and privileges; nothing on + macOS/Windows without a kernel extension; impossible in wasm. +- Faults are synchronous and uninterruptible from the parser's view: a network + error becomes `SIGBUS`, a stalled request hangs the thread. +- No batching: raw data arrives one fault (4 KiB, or the FUSE read size) at a + time unless the handler guesses read-ahead. +- 32-bit address space is still exhausted by mapping a large file. + +Rejected. It is how one would retrofit a C library that cannot change; we can. + +### How other readers do it + +- **libhdf5 `ros3`** replaces each POSIX read with an S3 range GET + ([h5py docs][h5py-file], [HDF Group, cloud storage options][hdfg-cloud]). + libhdf5 has a real metadata cache above the driver, and HDF5 2.2.0 added an + I/O block cache to ros3 "to reduce the number of requests to S3 for files + not using paged allocation", while paged files use the page buffer + ([HDF5 2.2.0 release][hdf5-220], [ros3 issue #4700][ros3-4700]). The page + buffer (`page_buf_size`) is "only allowed for HDF5 files created with + fs_strategy='page'" ([h5py docs][h5py-file]). PyPI h5py wheels do not + include ros3. +- **h5py + fsspec** hands libhdf5 a Python file object; each libhdf5 read + becomes a `read()` on an fsspec file, whose cache decides the requests. + fsspec's old default `readahead` cache requested 16x more data than + `blockcache` when opening HDF5 files; earthaccess now defaults to + `blockcache` with 4–16 MiB blocks by file size ([earthaccess][ea]); the + Cloud-Optimized guide recommends `blockcache` plus h5py's `page_buf_size` + and `rdcc_nbytes` ([guide][cog]). Our §2 comparison uses exactly this + path, without fsspec's cache. +- **pyfive** (pure Python, like us) supports lazy loading "on both Posix and + S3 filesystems" through fsspec, reads a variable's attributes and chunk + B-tree when it is accessed, and can merge chunk range requests with + fsspec's `merge_range_requests` ([pyfive][pyfive], [pyfive #257][pyfive-257]). +- **h5wasm** (libhdf5 compiled with Emscripten) backs a file with + `FS.createLazyFile`, which "can use range requests to incrementally access + the h5 file over the wire" ([h5wasm][h5wasm]); Emscripten's lazy files use + synchronous XHR, which browsers only allow in Web Workers + ([Emscripten FS API][emfs]). Files larger than memory remain an open issue + ([h5wasm #40][h5wasm-40]). +- **jsfive** (pure JS port of pyfive) reads from an `ArrayBuffer` of the whole + file ([jsfive][jsfive]); no lazy loading. +- **Rust `object_store`** (0.14): one `ObjectStore` trait over S3, GCS, Azure, + HTTP/WebDAV, local files and memory; `get_ranges` "will automatically + coalesce adjacent ranges into an appropriate number of parallel requests", + with `OBJECT_STORE_COALESCE_DEFAULT` as the gap below which ranges merge + ([docs.rs][os]). It builds for wasm32 except the local-filesystem and + some chunked-upload parts. + +## 4. Recommendation and migration plan + +Adopt **(a)**, with a block cache as a required part of every non-local +backend, **(c)** as a cache policy, and the wasm path through the restartable +`NeedBytes` mode. Every milestone keeps `main` green: `cargo test +--workspace`, clippy, the conformance gate at 575/697 unchanged, and the mmap +fast path within benchmark noise. + +**M0 — prerequisites (≈1 week).** +- Dense-group name lookup through the v2 B-tree name index (Jenkins hash, + record type 5), instead of enumerating all links; same for creation-order + and attribute name lookups. §2 shows 936 ranges → a handful. Regression + test: count links decoded for one lookup in a 35 001-link group. +- One checked helper for address → index conversion; replace the 133 + `as usize` casts. On 32-bit an address past `usize::MAX` is a clean error + (today it truncates or panics). +- `Group::dataset(name)`/`Group::group(name)` call `children()`, which + re-reads the group's whole link list on every lookup, so walking a group of + n children decodes its links O(n) times. Look names up through the index + (above) and let a listing hand out its entries, so the cache has less to + absorb. + +**M1 — metadata over the trait, in-memory impl identical to today (2–3 weeks).** +- Add `Storage` (above) to `clawhdf5-format`, `no_std`-compatible, with + `impl Storage for [u8]`. Add a storage error variant — `FormatError` and the + facade `Error` are not `#[non_exhaustive]`, so this is a breaking change for + exhaustive matches: bundle it with the next major version, or mark both + enums `#[non_exhaustive]` first. +- Convert modules bottom-up — superblock, object header, local/global heap, + B-tree v1/v2, fractal heap, fixed/extensible array, symbol table, group + v1/v2, shared messages, attributes, fill value, data layout — one commit + each. The old `&[u8]` signature stays as a thin wrapper over the new one + (`fn parse(data: &[u8], ..) { parse_in(data as &dyn Storage, ..) }`), so + callers and the other crates don't move yet. +- Replace the 5 open-ended slices and 38 `len()` checks with bounded reads. + +**M2 — raw data over the trait (1–2 weeks).** +- `data_read`, `chunked_read`, `parallel_read`, `partial_read`, `vds`, + VL/global heap. Chunked reads first collect the chunk extents the selection + needs, then call `read_ranges` once and decompress in parallel as today. +- Zero-copy stays: when `as_contiguous()` is `Some`, contiguous reads return + borrowed slices; `read_raw_ref`/`read_*_zerocopy`/`File::as_bytes` keep + their signatures and return a clear "not available for this storage" error + on other backends (they already return `Option`/`Result`). +- Facade: `File::open_storage(Box)`; `File::open` + keeps mmap and `from_bytes` keeps `Vec`, both through `impl Storage for [u8]`. + +**M3 — HTTP/S3 backend (1–2 weeks).** +- `clawhdf5-io`, feature `remote` (off by default, so the default tree stays + free of C and TLS stacks): `RangeStorage` over `object_store` (HTTP, S3, GCS, + Azure), with a `BlockCache` (LRU, block size configurable, default 1 MiB per + §2; the page size for paged files; the first block prefetched on open) and + a request counter exposed for tests and users. +- Python bindings: `clawhdf5.File("s3://…")` / `https://` through it. + +**M4 — wasm lazy loading (1–2 weeks).** +- `clawhdf5-wasm`: `openUrl(url) -> Promise` backed by `fetch` with a + `Range` header, on the main thread, via the restartable `NeedBytes` loop (no + Worker, no synchronous XHR — the thing h5wasm's lazy files need). Falls back + to a whole download when the server does not answer 206. +- `examples/wasm-viewer`: open by URL. + +**M5 — SWMR and growth (later, separate design).** `Storage::len()` may grow; +add `File::refresh()` that re-reads the superblock/EOF and invalidates cached +blocks past the old end. Needs libhdf5 SWMR semantics research first. + +Total: roughly 6–10 engineer-weeks for M0–M4 (estimate, not measured). + +### Keeping the local fast path + +- `impl Storage for [u8]` returns `Cow::Borrowed` — no copy, no allocation. +- Hot loops (raw-data copies, contiguous typed reads, `read_selection_native`) + branch once on `as_contiguous()` and then run today's code. +- `&dyn` dispatch is per structure, not per byte; parse code keeps working on + the returned slice. +- Gate: `crates/clawhdf5/benches/mmap_bench.rs`, the concurrent-read benches in + `clawhdf5-bench`, and the conformance run time, before and after each M1/M2 + commit, on an otherwise idle machine. Anything outside noise blocks the + commit. + +### Risks + +- **Silent regressions on local files** — mitigated by the bench gate above + and by M1 being a pure refactor (every conformance hash identical). +- **Two reads per structure** (prefix, then body) could double requests on a + cold cache. Block-aligned caching makes the second read a hit; the measured + block counts already include this pattern. +- **API break** — new error variant (see M1); `as_bytes()`-style APIs become + fallible for non-local storage. ClawBrainHub uses `File`, `FileBuilder`, + `AttrValue`, `Selection` on local files only, so it is unaffected by + behaviour, only by exhaustive matches on `Error`. +- **Restartable parsing** assumes operations are pure over the storage. The + chunk cache and metadata cache must only be filled by completed reads. +- **Cache memory** — the block cache needs a byte budget and eviction, which + (unlike option (b)) is sound because parsers hold `Cow`s, not borrows into + the cache. + +### Testing + +- **In-memory equivalence:** a `CountingStorage` wrapper over `[u8]` that + records every `read_at`; run the conformance probe through it and require + identical results. It also produces the request counts of §2 without the + trap-flag tracer. +- **Adversarial storage:** a wrapper that returns short reads, errors on the + Nth request, or serves blocks of 1 byte, to prove every miss is an `Err` + and never wrong data. +- **HTTP:** an in-process server on `127.0.0.1` (std `TcpListener`) that + honours `Range`, answers 206/416, can refuse ranges (200), and counts + requests. Tests assert both correctness against h5py and **request budgets** + (e.g. listing file A with a 1 MiB block cache takes ≤ 3 requests) so a + change that makes the reader chattier fails CI. +- **S3:** `object_store`'s in-memory store in unit tests; a MinIO or real + bucket only in an opt-in job. +- **wasm:** the existing Node/Chromium harness in `examples/wasm-viewer/test` + with a local range-capable server. +- **32-bit:** a `wasm32` or `i686` build that opens a sparse > 4 GiB file + through the counting storage. + +## Tools + +- `docs/design/tools/inventory.py` — §1 tables (`--list`, `--patterns`). +- `docs/design/tools/range-trace/` — §2 tracer (standalone crate, x86-64 + Linux, not part of the workspace). +- `docs/design/tools/libhdf5_reads.py` — §2 libhdf5 comparison and chunk + extents. + +[cog]: https://guide.cloudnativegeo.org/cloud-optimized-netcdf4-hdf5/ +[ea]: https://earthaccess.readthedocs.io/en/latest/user/explanation/fsspec/ +[h5py-file]: https://docs.h5py.org/en/stable/high/file.html +[hdfg-cloud]: https://www.hdfgroup.org/2022/08/08/cloud-storage-options-for-hdf5/ +[hdf5-220]: https://www.hdfgroup.org/2026/07/30/release-of-hdf5-2-2-0-and-two-august-events-newsletter-210/ +[ros3-4700]: https://github.com/HDFGroup/hdf5/issues/4700 +[pyfive]: https://pyfive.readthedocs.io/en/latest/quickstart/usage.html +[pyfive-257]: https://github.com/NCAS-CMS/pyfive/issues/257 +[h5wasm]: https://github.com/usnistgov/h5wasm +[h5wasm-40]: https://github.com/usnistgov/h5wasm/issues/40 +[emfs]: https://emscripten.org/docs/api_reference/Filesystem-API.html +[jsfive]: https://github.com/usnistgov/jsfive +[os]: https://docs.rs/object_store/latest/object_store/ diff --git a/docs/design/tools/inventory.py b/docs/design/tools/inventory.py new file mode 100644 index 0000000..ce405af --- /dev/null +++ b/docs/design/tools/inventory.py @@ -0,0 +1,131 @@ +#!/usr/bin/env python3 +"""Inventory of whole-file `&[u8]` parameters in the clawhdf5 workspace. + +Used by docs/design/range-reads.md. Run from the repository root: + + python3 docs/design/tools/inventory.py # per-file table + python3 docs/design/tools/inventory.py --list # every signature + python3 docs/design/tools/inventory.py --patterns # read patterns per crate + +A function counts as taking "the whole file" when it has a parameter named +`file_data: &[u8]` (the repository convention), or a `data` / `file` / `buf` / +`bytes` / `mmap` parameter of type `&[u8]` *together with* a parameter whose +name says it is a file address (`*address*`, `*addr*`, `*offset*` of an +integer type). The second rule is a heuristic; `--list` prints every match so +it can be checked by eye. Code after the first `#[cfg(test)] mod ... {` in a +file is excluded (the repository keeps unit tests at the end of each file), +as are the tests/ and benches/ directories. +""" +import os +import re +import sys +from collections import defaultdict + +ROOT = os.getcwd() +FN_RE = re.compile(r"\bfn\s+([A-Za-z_][A-Za-z0-9_]*)\s*(<[^()]*?>)?\s*\(", re.S) +PARAM_RE = re.compile(r"([A-Za-z_][A-Za-z0-9_]*)\s*:\s*&(?:'[a-z_]+\s+)?\[u8\]") +ADDR_RE = re.compile(r"\b([a-z_]*(?:address|addr|offset)[a-z_]*)\s*:\s*(?:u64|usize|u32)") +WHOLE_NAMES = {"data", "file", "buf", "bytes", "file_bytes", "mmap"} + + +def signature(src, start): + depth, i = 0, start + while i < len(src): + c = src[i] + if c == "(": + depth += 1 + elif c == ")": + depth -= 1 + if depth == 0: + return src[start + 1 : i] + i += 1 + return "" + + +def non_test_source(src): + m = re.search(r"#\[cfg\(test\)\]\s*mod\s+\w+\s*\{", src) + return src[: m.start()] if m else src + + +PATTERNS = [ + ("`file_data` passed on (call sites)", re.compile(r"[(,]\s*&?file_data\s*[,)]")), + ("`file_data[..]` slicing", re.compile(r"\bfile_data\s*\[")), + ("open-ended `file_data[x..]`", re.compile(r"\bfile_data\s*\[[^\]]*\.\.\s*\]")), + ("`file_data.get(..)`", re.compile(r"\bfile_data\s*\.\s*get\s*\(")), + ("`file_data.len()`", re.compile(r"\bfile_data\s*\.\s*len\s*\(\)")), + ("address/offset `as usize` casts", re.compile(r"\b[a-z_]*(?:addr|address|offset)[a-z_]*\s+as\s+usize")), + ("`ObjectHeader::parse(` calls", re.compile(r"ObjectHeader::parse\s*\(")), + ("`.as_bytes()` on a file/reader", re.compile(r"\b(?:file|reader|data|self\.file|self\.data|self\.file\.data|root\.file)\s*\.\s*as_bytes\s*\(\)")), +] + + +def patterns(): + """Per-crate counts of the read patterns (non-test code only).""" + per_crate = defaultdict(lambda: [0] * len(PATTERNS)) + for crate in sorted(os.listdir(os.path.join(ROOT, "crates"))): + srcdir = os.path.join(ROOT, "crates", crate, "src") + for dp, _, fns in os.walk(srcdir): + for fn in fns: + if not fn.endswith(".rs"): + continue + with open(os.path.join(dp, fn), encoding="utf-8") as fh: + src = non_test_source(fh.read()) + for i, (_, rx) in enumerate(PATTERNS): + per_crate[crate][i] += len(rx.findall(src)) + print("| crate | " + " | ".join(n for n, _ in PATTERNS) + " |") + print("|---|" + "---:|" * len(PATTERNS)) + tot = [0] * len(PATTERNS) + for c, v in sorted(per_crate.items()): + if any(v): + print("| %s | %s |" % (c, " | ".join(str(x) for x in v))) + tot = [a + b for a, b in zip(tot, v)] + print("| **total** | %s |" % " | ".join("**%d**" % x for x in tot)) + + +def main(): + if "--patterns" in sys.argv: + patterns() + return + per_file = defaultdict(lambda: [0, 0, 0]) # [file_data, heuristic, pub] + rows = [] + for crate in sorted(os.listdir(os.path.join(ROOT, "crates"))): + srcdir = os.path.join(ROOT, "crates", crate, "src") + for dp, _, fns in os.walk(srcdir): + for fn in sorted(fns): + if not fn.endswith(".rs"): + continue + path = os.path.join(dp, fn) + with open(path, encoding="utf-8") as fh: + src = non_test_source(fh.read()) + for m in FN_RE.finditer(src): + sig = signature(src, m.end() - 1) + params = PARAM_RE.findall(sig) + kind = None + if "file_data" in params: + kind = "file_data" + elif any(p in WHOLE_NAMES for p in params) and ADDR_RE.search(sig): + kind = "heuristic" + if not kind: + continue + rel = os.path.relpath(path, ROOT) + line_start = src.rfind("\n", 0, m.start()) + 1 + is_pub = src[line_start : m.start()].strip().startswith("pub") + per_file[rel][0 if kind == "file_data" else 1] += 1 + per_file[rel][2] += int(is_pub) + line = src.count("\n", 0, m.start()) + 1 + rows.append((rel, line, m.group(1), kind, is_pub)) + if "--list" in sys.argv: + for r in rows: + print("%s:%d %s [%s%s]" % (r[0], r[1], r[2], r[3], ", pub" if r[4] else "")) + return + print("| file | `file_data` fns | other whole-slice fns (heuristic) | of which `pub` |") + print("|---|---:|---:|---:|") + tot = [0, 0, 0] + for f, (a, b, p) in sorted(per_file.items(), key=lambda kv: (-(kv[1][0] + kv[1][1]), kv[0])): + print("| %s | %d | %d | %d |" % (f, a, b, p)) + tot = [tot[0] + a, tot[1] + b, tot[2] + p] + print("| **total** | **%d** | **%d** | **%d** |" % tuple(tot)) + + +if __name__ == "__main__": + main() diff --git a/docs/design/tools/libhdf5_reads.py b/docs/design/tools/libhdf5_reads.py new file mode 100644 index 0000000..5d6053e --- /dev/null +++ b/docs/design/tools/libhdf5_reads.py @@ -0,0 +1,98 @@ +#!/usr/bin/env python3 +"""The libhdf5 side of the range-read measurement (docs/design/range-reads.md). + + libhdf5_reads.py FILE DATASET # count libhdf5's reads + libhdf5_reads.py FILE DATASET --extents # print DATASET's stored extents + +Counting: the file is opened through h5py's `fileobj` driver with a Python +file-like object that logs every `readinto`/`read` libhdf5 makes (this is how +h5py + fsspec reads remote files: each call becomes a range request unless +fsspec's own block cache absorbs it). The phases mirror range-trace: open, +list (visit every object; shape and dtype of every dataset), read (the whole +dataset). Default h5py settings (libhdf5's metadata cache, 64 KiB sieve +buffer, 1 MiB raw chunk cache) apply. + +--extents prints `offset size` lines for the dataset's stored data (the +contiguous block, or every allocated chunk), the input range-trace uses to +split its read phase into metadata and raw data. +""" +import sys + +import h5py + + +class LoggingFile: + def __init__(self, path): + self.f = open(path, "rb") + self.pos = 0 + self.log = [] + + def seek(self, off, whence=0): + self.pos = self.f.seek(off, whence) + return self.pos + + def tell(self): + return self.pos + + def readinto(self, b): + n = self.f.readinto(b) + self.log.append((self.pos, n)) + self.pos += n + return n + + def read(self, size=-1): + data = self.f.read(size) + self.log.append((self.pos, len(data))) + self.pos += len(data) + return data + + +def extents(path, name): + with h5py.File(path, "r") as f: + ds = f[name] + if ds.chunks is None: + off = ds.id.get_offset() + if off is not None: + print(off, ds.id.get_storage_size()) + return + for i in range(ds.id.get_num_chunks()): + info = ds.id.get_chunk_info(i) + print(info.byte_offset, info.size) + + +def summarise(label, log): + n = len(log) + total = sum(s for _, s in log) + distinct = len(set(log)) + print("| %s | %d | %d | %d |" % (label, n, distinct, total)) + + +def count(path, name): + lf = LoggingFile(path) + f = h5py.File(lf, "r") + opened = list(lf.log) + lf.log.clear() + + def visit(_n, obj): + if isinstance(obj, h5py.Dataset): + obj.shape, obj.dtype + + f.visititems(visit) + listed = list(lf.log) + lf.log.clear() + f[name][()] + readlog = list(lf.log) + f.close() + print("| phase | read calls | distinct (offset, len) | bytes |") + print("|---|---:|---:|---:|") + summarise("open", opened) + summarise("list", listed) + summarise("read", readlog) + summarise("open+list+read", opened + listed + readlog) + + +if __name__ == "__main__": + if len(sys.argv) >= 4 and sys.argv[3] == "--extents": + extents(sys.argv[1], sys.argv[2]) + else: + count(sys.argv[1], sys.argv[2]) diff --git a/docs/design/tools/range-trace/Cargo.toml b/docs/design/tools/range-trace/Cargo.toml new file mode 100644 index 0000000..5b0d8cd --- /dev/null +++ b/docs/design/tools/range-trace/Cargo.toml @@ -0,0 +1,17 @@ +[package] +name = "range-trace" +version = "0.1.0" +edition = "2024" +rust-version = "1.92" +publish = false +description = "Records which byte ranges of an HDF5 file the clawhdf5 facade reads (docs/design/range-reads.md)" + +# Outside the main workspace on purpose: a measurement tool for a design +# document, never built by `cargo test --workspace`. x86-64 Linux only. +[workspace] + +[dependencies] +# Default features minus nothing: `parallel` is off by default, and the tracer +# relies on every read happening on the main thread. +clawhdf5 = { path = "../../../../crates/clawhdf5" } +libc = "0.2" diff --git a/docs/design/tools/range-trace/src/main.rs b/docs/design/tools/range-trace/src/main.rs new file mode 100644 index 0000000..24b7ce8 --- /dev/null +++ b/docs/design/tools/range-trace/src/main.rs @@ -0,0 +1,375 @@ +//! range-trace: which bytes of an HDF5 file does clawhdf5 read? +//! +//! Measurement tool for `docs/design/range-reads.md` (x86-64 Linux only). +//! +//! The file is loaded into a page-aligned buffer and handed to +//! `clawhdf5::File::from_bytes`, so the library parses it exactly as it does +//! today. The buffer is then `mprotect`ed to `PROT_NONE`. Every load from it +//! faults; the SIGSEGV handler logs the exact faulting address, makes that +//! page readable and sets the x86 trap flag, so the CPU single-steps the one +//! instruction and the SIGTRAP handler re-protects the page. Every load +//! instruction that touches the file is therefore recorded (with its first +//! byte; widths are not decoded, see `ACCESS_WIDTH`). +//! +//! Bulk copies (raw data) would fault once per load; a page that faults more +//! than `BULK_THRESHOLD` times in one phase is left readable for the rest of +//! that phase and counted as wholly read ("bulk page"). +//! +//! Phases: `open` (superblock), `list` (walk every group; for every dataset +//! its shape and dtype, i.e. what `h5ls -r -v` or a tree view needs), and +//! `read` (read the named dataset in full through `read_selection(All)`). +//! +//! Usage: range-trace FILE DATASET_PATH [RAW_EXTENTS] +//! +//! RAW_EXTENTS (optional) lists the dataset's stored data as `offset size` +//! lines (absolute file offsets; `libhdf5_reads.py --extents` writes it from +//! h5py). With it the `read` phase is split into `read (metadata)`, the +//! loads outside those extents, and `read (raw data)`, the loads inside. +//! +//! Every read must happen on this thread: build without the facade's +//! `parallel` feature (off by default). + +use std::alloc::{Layout, alloc_zeroed}; +use std::collections::BTreeSet; +use std::sync::atomic::{AtomicPtr, AtomicUsize, Ordering}; + +use clawhdf5::{File, Group, Selection}; + +const PAGE: usize = 4096; +const BULK_THRESHOLD: u32 = 4096; +/// Bytes assumed read by one logged load (the widest scalar load the parsers +/// issue; SIMD copies are bulk anyway). +const ACCESS_WIDTH: u64 = 8; +/// Two accesses closer than this belong to the same structure, i.e. would be +/// one range request. +const MERGE_GAP: u64 = 64; +const LOG_CAP: usize = 64 << 20; + +static BUF_START: AtomicUsize = AtomicUsize::new(0); +static BUF_LEN: AtomicUsize = AtomicUsize::new(0); +static LOG: AtomicPtr = AtomicPtr::new(std::ptr::null_mut()); +static LOG_LEN: AtomicUsize = AtomicUsize::new(0); +static HITS: AtomicPtr = AtomicPtr::new(std::ptr::null_mut()); +static BULK: AtomicPtr = AtomicPtr::new(std::ptr::null_mut()); +static PENDING: [AtomicUsize; 8] = [const { AtomicUsize::new(0) }; 8]; +static PENDING_N: AtomicUsize = AtomicUsize::new(0); + +const TF: i64 = 0x100; + +extern "C" fn on_segv(_sig: libc::c_int, info: *mut libc::siginfo_t, ctx: *mut libc::c_void) { + unsafe { + let addr = (*info).si_addr() as usize; + let start = BUF_START.load(Ordering::Relaxed); + let len = BUF_LEN.load(Ordering::Relaxed); + if addr < start || addr >= start + len { + // A genuine crash: restore the default action and re-fault. + libc::signal(libc::SIGSEGV, libc::SIG_DFL); + return; + } + let off = addr - start; + let n = LOG_LEN.load(Ordering::Relaxed); + if n < LOG_CAP { + *LOG.load(Ordering::Relaxed).add(n) = off as u64; + LOG_LEN.store(n + 1, Ordering::Relaxed); + } + let page = off / PAGE; + let hits = HITS.load(Ordering::Relaxed).add(page); + *hits += 1; + libc::mprotect( + (start + page * PAGE) as *mut libc::c_void, + PAGE, + libc::PROT_READ | libc::PROT_WRITE, + ); + if *hits >= BULK_THRESHOLD { + *BULK.load(Ordering::Relaxed).add(page) = 1; + } else { + let p = PENDING_N.load(Ordering::Relaxed); + if p < PENDING.len() { + PENDING[p].store(page, Ordering::Relaxed); + PENDING_N.store(p + 1, Ordering::Relaxed); + } + } + let uc = ctx as *mut libc::ucontext_t; + (*uc).uc_mcontext.gregs[libc::REG_EFL as usize] |= TF; + } +} + +extern "C" fn on_trap(_sig: libc::c_int, _info: *mut libc::siginfo_t, ctx: *mut libc::c_void) { + unsafe { + let start = BUF_START.load(Ordering::Relaxed); + let n = PENDING_N.load(Ordering::Relaxed); + for p in PENDING.iter().take(n) { + let page = p.load(Ordering::Relaxed); + libc::mprotect( + (start + page * PAGE) as *mut libc::c_void, + PAGE, + libc::PROT_NONE, + ); + } + PENDING_N.store(0, Ordering::Relaxed); + let uc = ctx as *mut libc::ucontext_t; + (*uc).uc_mcontext.gregs[libc::REG_EFL as usize] &= !TF; + } +} + +fn install( + sig: libc::c_int, + h: extern "C" fn(libc::c_int, *mut libc::siginfo_t, *mut libc::c_void), +) { + unsafe { + let mut sa: libc::sigaction = std::mem::zeroed(); + sa.sa_sigaction = h as usize; + sa.sa_flags = libc::SA_SIGINFO | libc::SA_NODEFER; + libc::sigemptyset(&mut sa.sa_mask); + assert_eq!(libc::sigaction(sig, &sa, std::ptr::null_mut()), 0); + } +} + +fn protect(prot: libc::c_int) { + let start = BUF_START.load(Ordering::Relaxed); + let len = BUF_LEN.load(Ordering::Relaxed); + unsafe { + assert_eq!(libc::mprotect(start as *mut libc::c_void, len, prot), 0); + } +} + +struct Phase { + name: &'static str, + log: Vec, + bulk_pages: Vec, +} + +/// Start a phase: clear the per-page state and protect the buffer. +fn begin() { + let pages = BUF_LEN.load(Ordering::Relaxed) / PAGE; + unsafe { + std::ptr::write_bytes(HITS.load(Ordering::Relaxed), 0, pages); + std::ptr::write_bytes(BULK.load(Ordering::Relaxed), 0, pages); + } + LOG_LEN.store(0, Ordering::Relaxed); + protect(libc::PROT_NONE); +} + +fn end(name: &'static str) -> Phase { + protect(libc::PROT_READ | libc::PROT_WRITE); + let n = LOG_LEN.load(Ordering::Relaxed); + let log = unsafe { std::slice::from_raw_parts(LOG.load(Ordering::Relaxed), n) }.to_vec(); + let pages = BUF_LEN.load(Ordering::Relaxed) / PAGE; + let bulk = unsafe { std::slice::from_raw_parts(BULK.load(Ordering::Relaxed), pages) }; + let bulk_pages = (0..pages) + .filter(|&p| bulk[p] != 0) + .map(|p| p as u64) + .collect(); + if n == LOG_CAP { + eprintln!("warning: access log full in phase {name}"); + } + Phase { + name, + log, + bulk_pages, + } +} + +/// Byte intervals [lo, hi) read in a phase, merged when closer than `gap`. +fn ranges(ph: &[&Phase], gap: u64) -> Vec<(u64, u64)> { + let mut iv: Vec<(u64, u64)> = Vec::new(); + for p in ph { + iv.extend(p.log.iter().map(|&o| (o, o + ACCESS_WIDTH))); + iv.extend( + p.bulk_pages + .iter() + .map(|&pg| (pg * PAGE as u64, (pg + 1) * PAGE as u64)), + ); + } + iv.sort_unstable(); + let mut out: Vec<(u64, u64)> = Vec::new(); + for (lo, hi) in iv { + match out.last_mut() { + Some(last) if lo <= last.1 + gap => last.1 = last.1.max(hi), + _ => out.push((lo, hi)), + } + } + out +} + +/// Requests a reader with no cache at all would make: a new request each +/// time the access stream leaves the neighbourhood of the current run. +fn uncached_requests(p: &Phase) -> usize { + let mut n = 0; + let (mut lo, mut hi) = (u64::MAX, 0u64); + for &o in &p.log { + if lo != u64::MAX && o + MERGE_GAP >= lo && o <= hi + MERGE_GAP { + lo = lo.min(o); + hi = hi.max(o + ACCESS_WIDTH); + } else { + n += 1; + lo = o; + hi = o + ACCESS_WIDTH; + } + } + n + p.bulk_pages.len() +} + +fn blocks(ph: &[&Phase], block: u64) -> usize { + let mut set = BTreeSet::new(); + for (lo, hi) in ranges(ph, 0) { + for b in lo / block..=(hi - 1) / block { + set.insert(b); + } + } + set.len() +} + +fn read_extents(path: &str) -> Vec<(u64, u64)> { + let text = std::fs::read_to_string(path).expect("read extents"); + let mut v: Vec<(u64, u64)> = text + .lines() + .filter_map(|l| { + let mut it = l.split_whitespace().map(|t| t.parse::()); + match (it.next(), it.next()) { + (Some(Ok(o)), Some(Ok(n))) => Some((o, o + n)), + _ => None, + } + }) + .collect(); + v.sort_unstable(); + v +} + +fn in_extents(ext: &[(u64, u64)], lo: u64, hi: u64) -> bool { + let i = ext.partition_point(|e| e.1 <= lo); + i < ext.len() && ext[i].0 < hi +} + +/// Split a phase into loads outside and inside the raw-data extents. A bulk +/// page counts as raw data when it overlaps an extent. +fn split_raw(p: &Phase, ext: &[(u64, u64)]) -> (Phase, Phase) { + let (mut m, mut r) = (Vec::new(), Vec::new()); + for &o in &p.log { + if in_extents(ext, o, o + 1) { + r.push(o) + } else { + m.push(o) + } + } + let pg = PAGE as u64; + let (bm, br): (Vec, Vec) = p + .bulk_pages + .iter() + .partition(|&&b| !in_extents(ext, b * pg, (b + 1) * pg)); + ( + Phase { + name: "read (metadata)", + log: m, + bulk_pages: bm, + }, + Phase { + name: "read (raw data)", + log: r, + bulk_pages: br, + }, + ) +} + +fn walk(g: &Group<'_>, path: &str, objs: &mut usize) { + for name in g.datasets().unwrap_or_default() { + *objs += 1; + if let Ok(ds) = g.dataset(&name) { + let _ = ds.shape(); + let _ = ds.dtype(); + } + } + for name in g.groups().unwrap_or_default() { + *objs += 1; + if let Ok(sub) = g.group(&name) { + walk(&sub, &format!("{path}/{name}"), objs); + } + } +} + +fn main() { + let args: Vec = std::env::args().collect(); + if args.len() != 3 && args.len() != 4 { + eprintln!("usage: range-trace FILE DATASET_PATH [RAW_EXTENTS]"); + std::process::exit(2); + } + let bytes = std::fs::read(&args[1]).expect("read file"); + let len = bytes.len(); + let cap = len.div_ceil(PAGE).max(1) * PAGE; + let pages = cap / PAGE; + // Page-aligned buffer so that protection covers exactly the file. The + // Vec is never dropped (its layout differs from Vec's own), see the end. + let ptr = unsafe { alloc_zeroed(Layout::from_size_align(cap, PAGE).unwrap()) }; + unsafe { std::ptr::copy_nonoverlapping(bytes.as_ptr(), ptr, len) }; + drop(bytes); + let buf = unsafe { Vec::from_raw_parts(ptr, len, cap) }; + BUF_START.store(ptr as usize, Ordering::Relaxed); + BUF_LEN.store(cap, Ordering::Relaxed); + let mut log = vec![0u64; LOG_CAP]; + LOG.store(log.as_mut_ptr(), Ordering::Relaxed); + let mut hits = vec![0u32; pages]; + HITS.store(hits.as_mut_ptr(), Ordering::Relaxed); + let mut bulk = vec![0u8; pages]; + BULK.store(bulk.as_mut_ptr(), Ordering::Relaxed); + install(libc::SIGSEGV, on_segv); + install(libc::SIGTRAP, on_trap); + + begin(); + let file = File::from_bytes(buf).expect("open"); + let p_open = end("open"); + + begin(); + let mut objs = 0; + walk(&file.root(), "", &mut objs); + let p_list = end("list"); + + begin(); + let out = file + .dataset(&args[2]) + .and_then(|d| d.read_selection(&Selection::All)) + .expect("read dataset"); + let p_read = end("read"); + + println!("file: {} ({} bytes), objects listed: {objs}", args[1], len); + println!("dataset: {} ({} bytes decoded)", args[2], out.len()); + println!( + "| phase | loads logged | bulk 4K pages | uncached requests | distinct ranges (gap<{MERGE_GAP}B) | bytes in ranges | 4 KiB blocks | 64 KiB blocks | 1 MiB blocks |" + ); + println!("|---|---:|---:|---:|---:|---:|---:|---:|---:|"); + let row = |label: String, ph: &[&Phase], uncached: usize| { + let r = ranges(ph, MERGE_GAP); + let bytes: u64 = r.iter().map(|(a, b)| b - a).sum(); + let loads: usize = ph.iter().map(|p| p.log.len()).sum(); + let bulk: usize = ph.iter().map(|p| p.bulk_pages.len()).sum(); + println!( + "| {label} | {loads} | {bulk} | {uncached} | {} | {bytes} | {} | {} | {} |", + r.len(), + blocks(ph, 4 << 10), + blocks(ph, 64 << 10), + blocks(ph, 1 << 20) + ); + }; + for p in [&p_open, &p_list, &p_read] { + row(p.name.to_string(), &[p], uncached_requests(p)); + } + if let Some(path) = args.get(3) { + let (meta, raw) = split_raw(&p_read, &read_extents(path)); + for p in [&meta, &raw] { + row(p.name.to_string(), &[p], uncached_requests(p)); + } + let all = [&p_open, &p_list, &meta]; + let unc: usize = all.iter().map(|p| uncached_requests(p)).sum(); + row("all metadata".into(), &all, unc); + } + let all = [&p_open, &p_list, &p_read]; + let unc: usize = all.iter().map(|p| uncached_requests(p)).sum(); + row("open+list+read".into(), &all, unc); + let meta = [&p_open, &p_list]; + let unc: usize = meta.iter().map(|p| uncached_requests(p)).sum(); + row("open+list".into(), &meta, unc); + // The File owns a buffer whose layout Vec does not know; never drop it. + std::mem::forget(file); + std::mem::forget(log); + std::mem::forget(hits); + std::mem::forget(bulk); +}