quantumclaw/clawhdf5
3
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

Chunked reads beat an h5py process pool; unlimited writer B-trees; Blosc2; 599/697 conformance #16

Merged
osobh merged 48 commits from feat/p2b-scale into main 2026-09-26 17:42:16 +00:00
Conversation 0 Commits 48 Files Changed 115
+1120
5 changed files with 1120 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Showing only changes of commit d0e3beb3aa - Show all commits
docs/design/range-reads.md
+499
View File
@@ -0,0 +1,499 @@
# Design: range reads (reading HDF5 without holding the whole file)
Status: proposal, 2026-09-26. No library code has changed; this document is
the plan for Phase 3's largest architectural change. Every count below was
taken on `tank` on 2026-09-26 at commit `de2a53f`, with the commands given
next to it. No timing numbers appear here on purpose: the machine was shared
with other build jobs when this was written.
## The problem
Every reader in clawhdf5 parses the file through one `&[u8]` that covers the
whole file:
- `clawhdf5_io::HDF5Read::as_bytes(&self) -> &[u8]` is the only read method of
the I/O trait (`crates/clawhdf5-io/src/lib.rs`). `FileReader` reads the
whole file into a `Vec`, `MmapReader` maps it.
- The facade's `File` keeps a `Backing` (`Vec<u8>` or mmap) and hands
`FileData::as_bytes()` — the file from the superblock to the recorded end of
file — to every `clawhdf5-format` call (`crates/clawhdf5/src/reader.rs`).
- The format crate's parsers take `file_data: &[u8]` plus a `usize` address and
slice into it.
- `LazyFile` (facade) parses metadata lazily, but still over
`HDF5Read::as_bytes`. `AsyncHDF5Read` (`clawhdf5-io`, `async` feature)
already has `read_at(offset, len)`, but `AsyncHDF5File::open` calls
`read_all()` and parses the result.
- The browser reader (`clawhdf5-wasm`) is `open(bytes: Vec<u8>)`: the page
must download the whole file before it can list a group.
That rules out four things users of other HDF5 readers have:
1. **Remote files by range request** — libhdf5's `ros3` driver, h5py over
`fsspec`/`s3fs`, pyfive over `fsspec`. Today a 30 GB file on S3 has to be
downloaded before its tree can be listed.
2. **Lazy loading in the browser** — h5wasm can back a file with HTTP range
requests; our wasm reader holds the whole file.
3. **SWMR readers of a growing file** — the slice is fixed at open; a reader
cannot see the file grow (and an mmap of a growing file is fragile).
4. **Files larger than the address space on 32-bit targets** — including
`wasm32`, where `usize` is 32 bits and a file above 4 GiB cannot even be put
in a slice; browsers cap `ArrayBuffer`s lower than that.
## 1. Inventory
### 1.1 Functions that take the whole file
`python3 docs/design/tools/inventory.py` scans non-test source (code before a
file's `#[cfg(test)] mod`, no `tests/`) for functions with a
`file_data: &[u8]` parameter (the repository's convention), and — as a
heuristic — functions with a `data`/`buf`/`file`/`bytes` slice *and* an
integer address/offset parameter. `--list` prints every hit.
- **103 functions take `file_data: &[u8]`** (73 of them `pub`), in 25 files.
- The heuristic adds 27, of which only **6 really take the whole file under
another name**: `ObjectHeader::parse` and its internal `parse_v1`,
`parse_v1_chunk`, `parse_v2`, `parse_v2_continuation` (`data: &[u8]`), and
`Superblock::parse`. The other 21 are helpers (`ensure_len`,
`read_length`, `read_uint`, …) that bound-check whatever slice they are
given, or writer-side `write_at`s in `clawhdf5-io`.
So **109 functions** have to change signature. Grouped by what they read:
| Read pattern | Modules (whole-file functions) | Count |
|---|---|---:|
| **Small metadata reads at addresses** — a structure whose size is usually only known after its prefix is read | `group_v2` 9, `shared_message` 9, `fractal_heap` 7, `attribute` 6, `object_header` 5, `group_v1` 5, `extensible_array` 4, `btree_v2` 4, `btree_v1` 3, `fixed_array` 3, `local_heap` 3, `fill_value` 3, `superblock` 2, `data_layout` 1, `symbol_table` 1 | 65 |
| **Bulk raw data** — a contiguous extent, or a list of chunk extents known once the chunk index has been walked | `chunked_read` 12, `data_read` 9, `vds` 6, `parallel_read` 3, `partial_read` 1, `provenance` 1 | 32 |
| **Heap-resident data** — VL strings and sequences: many small reads into global-heap collections | `vl_data` 3, `global_heap` 2, facade `vlen.rs` 3, facade `types.rs` 3 | 11 |
| Bindings | `clawhdf5-py/src/convert.rs` | 1 |
### 1.2 How they read
`python3 docs/design/tools/inventory.py --patterns` (non-test code):
| Pattern | Where | Count |
|---|---|---:|
| `file_data` passed on to another function (call sites) | format 262, facade 7, py 2 | 271 |
| `file_data[..]` slicing | format | 137 |
| of which open-ended `file_data[x..]` (slice to end of file) | format | 5 |
| `file_data.len()` (end-of-file bound checks) | format | 38 |
| address/offset `as usize` casts | format 120, facade 5, io 5, ann 3 | 133 |
| `.as_bytes()` on a file or reader, feeding format calls | facade 26, py 8, tools 1, wasm 1 | 36 |
Three things in this table shape the design:
- **Open-ended slices and `len()` checks** assume the whole file is present.
With a range reader "the rest of the file" is a request for gigabytes; each
of these 43 sites must become a bounded read.
- **The 133 `as usize` casts** are where a 64-bit HDF5 address is squeezed
into a pointer-sized index. On 32-bit targets they truncate or must fail.
A storage trait that takes `u64` offsets removes most of them.
- **The bytes escape through the public API.** `File::as_bytes()`,
`LazyFile::as_bytes()`, `MmapFile::as_bytes()` return the whole file, and 15
facade methods return `&'f [u8]`/`&'f [T]` borrowed from it
(`Dataset::read_raw_ref`, `read_as_slice`, `read_*_zerocopy`). The Python
bindings (`clawhdf5-py/src/node.rs`, `attrs.rs`, …) call
`clawhdf5_format` directly with `file.as_bytes()`, and so do `h5rs` and the
wasm reader. These must keep working for local files (they are the
zero-copy fast path) and fail cleanly for remote ones.
### 1.3 Pieces that already exist
- `clawhdf5_format::metadata_cache::MetadataCache` — LRU keyed by file offset
with a byte budget (2 MiB default). Not currently in the read path.
- `clawhdf5_io::prefetch::PrefetchReader` — ring buffer of `(offset, len)`
entries over an `HDF5Read`, plus a sweep detector for chunk access patterns.
- `clawhdf5_io::async_read::AsyncHDF5Read::read_at` — the right shape of
trait, but async and only used to read everything.
- `clawhdf5_io::hsds` — a client for the HSDS REST service. Different
protocol (the server parses HDF5), not a range reader.
## 2. Measurement: how chatty is a naive range reader?
`docs/design/tools/range-trace/` loads a file into a page-aligned buffer,
opens it with `clawhdf5::File::from_bytes` (today's code, unchanged),
`mprotect`s the buffer and single-steps every load that faults (SIGSEGV logs
the exact address and unprotects the page; the x86 trap flag re-protects it
after one instruction). So every load instruction that touches the file is
recorded, with no change to the library. A page loaded more than 4096 times in
a phase is left readable and counted as wholly read. Phases: **open**
(superblock), **list** (walk every group; shape and dtype of every dataset —
what a tree view or `h5ls -r -v` needs), **read** (one dataset, whole). With
the dataset's chunk extents (from h5py, `libhdf5_reads.py --extents`) the read
phase is split into the chunk index and the raw data.
Columns: *uncached requests* — a new request every time the access stream
leaves the neighbourhood (±64 B) of the current run, i.e. a reader that turns
each parse into a read with no cache; *ranges* — distinct byte ranges after
merging accesses closer than 64 B; *N KiB blocks* — distinct aligned blocks,
i.e. the requests a reader with a block cache of that size would make. For
comparison, `libhdf5_reads.py` opens the same file in h5py through a Python
file object (the `fileobj` driver — exactly how h5py + fsspec read remote
files) and counts libhdf5's `read` calls, default h5py settings.
```bash
CARGO_TARGET_DIR=$PWD/target cargo build --release \
--manifest-path docs/design/tools/range-trace/Cargo.toml
PY=/home/osobh/projects/clawhdf5/.venv/bin/python
$PY docs/design/tools/libhdf5_reads.py FILE DSET --extents > ext.txt
target/release/range-trace FILE DSET ext.txt
$PY docs/design/tools/libhdf5_reads.py FILE DSET
```
Files (from `conformance/.cache/corpus`), 2026-09-26 on tank:
- **A** `xarray-data/imerghh_730.hdf5` — NASA IMERG, 7.7 MB, superblock v0,
21 objects; read `/Grid/precipitation` (25 deflated chunks).
- **B** `NCAS-CMS_pyfive/tests/data/cmip_bad_eg.nc` — netCDF-4, 48 MB,
superblock v2, 9 objects; read `/tas` (780 deflated chunks, 43.7 MB stored).
- **C** `hdf5/tools/test/testfiles/h5stat_newgrat.h5` — 6.4 MB, superblock
v3, 35 001 groups in one dense (fractal heap + v2 B-tree) root group; read
`/DATASET_NAME` (a scalar with no storage).
- **B′** B repacked with paged aggregation:
`h5repack -S PAGE -G 65536` (55.5 MB).
clawhdf5 (range-trace):
| File | Phase | Uncached requests | Ranges | Bytes in ranges | 4 KiB blocks | 64 KiB blocks | 1 MiB blocks |
|---|---|---:|---:|---:|---:|---:|---:|
| A | open + list | 410 | 5 | 25 285 | 8 | 3 | 2 |
| A | read: chunk index | 33 | 8 | 4 608 | 7 | 3 | 2 |
| A | read: raw data | (25 chunks) | 2 | 1 496 533 | 368 | 25 | 3 |
| A | all metadata | 443 | 6 | 26 549 | 10 | 3 | 2 |
| B | open + list | 143 | 5 | 17 951 | 7 | 1 | 1 |
| B | read: chunk index | 46 | 19 | 49 583 | 28 | 15 | 14 |
| B | read: raw data | (780 chunks) | 15 | 43 656 557 | 10 671 | 668 | 43 |
| B | all metadata | 189 | 20 | 57 023 | 30 | 15 | 14 |
| B′ | all metadata | 413 | 24 | 71 083 | 25 | 3 | 2 |
| B′ | read: raw data | (780 chunks) | 780 | 43 613 717 | 10 920 | 780 | 49 |
| C | open + list | 455 779 | 3 | 6 357 627 | 1 553 | 98 | 7 |
| C | read: resolve `/DATASET_NAME` | 38 525 | 936 | 1 137 116 | 948 | 98 | 7 |
libhdf5 through h5py's `fileobj` driver (read calls, bytes):
| File | open | list | read | total calls | total bytes |
|---|---:|---:|---:|---:|---:|
| A | 3 | 93 | 25 | 121 | 1 563 559 |
| B | 2 | 78 | 780 | 860 | 43 769 952 |
| C | 4 | 36 105 | 1 | 36 110 | 19 725 823 |
What this says:
1. **A reader with no cache is hopeless.** clawhdf5's parsers revisit the same
structures many times (A: 26 000 loads to list 21 objects:
`Group::dataset(name)` and `Group::group(name)` re-read the group's whole
link list for every lookup). Mapped
one-to-one onto requests that is 410 round trips to list 21 objects, and
455 779 to list C. **A cache is not an optimisation, it is the design.**
2. **With a block cache, metadata is cheap on typical files.** Metadata of A
and B touches 5–20 ranges and 2–15 blocks of 1 MiB; a 1 MiB-block cache
lists A in 2 requests and B in 1, where libhdf5 issues 96 and 80 calls
(fsspec's block cache absorbs those for h5py).
3. **"Prefetch the metadata region" alone does not work.** In B the chunk
index (v1 B-tree nodes) is interleaved with the raw data: the metadata
touches 14 distinct 1 MiB blocks spread over 48 MB. Only 4 of the 611
corpus files h5py can open use paged aggregation (all four are libhdf5
test files; 502 have a v0 superblock). Repacked as B′, the same metadata
fits in 3 blocks of 64 KiB — paging helps a lot when present, but a reader
cannot count on it.
4. **Raw data is naturally a batch.** Once the chunk index is walked, all
chunk extents are known: B's 780 chunks are 15 byte ranges after merging
neighbours, so a coalescing `get_ranges` call reads the dataset in a
handful of parallel requests. libhdf5 issues one read per chunk (780).
5. **An existing inefficiency becomes a blocker.** Resolving `/DATASET_NAME`
in C reads 1.1 MB (936 ranges) because
`group_v2::resolve_path_following_links` enumerates every link of the
group (`resolve_group_entries`) and compares names, instead of hashing
the name and descending the v2 B-tree name index. On an mmap this is
merely slow; over HTTP it is 98 requests of 64 KiB for one lookup.
libhdf5 reads one 512-byte block. Listing C is inherently whole-file (35 001
object headers spread over the file), and libhdf5 reads 19.7 MB — three
times the file — to do it.
Caveats: loads are logged with their first byte and a nominal width of 8 B,
so byte totals are approximate (±64 B per range); "uncached requests" is a
model, not a measured reader. These are counts, not timings.
## 3. Options
### (a) A storage trait threaded through the format crate
```rust
// clawhdf5-format (no_std + alloc)
pub trait Storage {
/// Bytes [offset, offset + len). Short only at end of file.
fn read_at(&self, offset: u64, len: usize) -> Result<Cow<'_, [u8]>, FormatError>;
/// Current length; may grow between calls (SWMR).
fn len(&self) -> u64;
/// Batch read; backends coalesce and parallelise. Default: loop.
fn read_ranges(&self, ranges: &[Range<u64>]) -> Result<Vec<Cow<'_, [u8]>>, FormatError> { .. }
/// The whole file as one slice, when the backend has it (Vec, mmap).
fn as_contiguous(&self) -> Option<&[u8]> { None }
}
impl Storage for [u8] { /* Cow::Borrowed, as_contiguous = Some(self) */ }
```
Every `file_data: &[u8]` becomes `file: &dyn Storage` (or `&S` generic), and
every `file_data[a..b]` becomes `file.read_at(a, b - a)?`.
- **For:** explicit, sound, errors are values (a network error is an `Err`,
not a panic); works for any backend; `u64` offsets fix 32-bit; `len()` can
grow, which is what SWMR needs; `read_ranges` gives raw data its natural
batch shape; `impl Storage for [u8]` lets the migration go one module at a
time with `&[u8]` callers unchanged.
- **Against:** touches all 109 functions and ~400 slice/len/call sites
(§1.2); every parse that learns a structure's size from its prefix needs two
reads (prefix, then body) — cheap against a cache, but it must be written
that way. `Cow::Owned` results cost a copy per read for remote backends.
- **Sync vs async:** the format crate is `no_std` and synchronous and should
stay so (parsing is CPU work; an async parser would colour 109 functions and
their callers). Remote backends are async. Two bridges, both needed:
- native: a backend that blocks on its own runtime (`object_store` +
a private tokio runtime), used from ordinary threads;
- wasm (no blocking on the main thread): a **restartable** mode where the
cache returns `FormatError::NeedBytes { offset, len }` on a miss; an async
driver fetches the block and re-runs the (pure, idempotent) operation.
parquet-rs's async reader uses the same "fetch, then parse synchronously"
split. The measurement bounds the retries: one per missed block, i.e. 1–15
for A and B's metadata at 1 MiB blocks.
- **`&dyn` vs generic:** generics monomorphise 109 functions per backend
(binary size matters for wasm); `&dyn Storage` costs one indirect call per
structure read, negligible next to parsing. Hot raw-data loops keep their
speed through `as_contiguous()`.
### (b) A page-cache "virtual slice"
A type that implements `Index<Range<usize>, Output = [u8]>` and faults pages
in on demand from interior-mutable storage, so parsers keep their slicing
syntax.
- `Index::index` returns `&[u8]` borrowed from `&self`. A cache that evicts
pages invalidates references that are still alive — unsound — so pages can
never be evicted while the object lives (unbounded memory, which is the
problem we are solving).
- A range that crosses a page boundary has to be copied into a contiguous
buffer that also must outlive the borrow: an append-only arena that only
grows.
- `Index` cannot fail: a network error or truncated response becomes a
panic, which breaks "never return wrong data; unsupported is a clean
error".
- Open-ended slices (`file_data[x..]`) and `file_data.len()` still mean "the
whole rest of the file".
- It hides the cost model: a `for` loop over `file_data[i]` becomes thousands
of cache lookups, and nothing in the signature says the call can block.
Rejected. It saves the signature churn of (a) but moves every failure into
panics and unsoundness.
### (c) Keep `&[u8]` for metadata by prefetching it; range-read raw data only
Fetch "the metadata" up front into a buffer, parse it with today's code, and
range-read only chunk data.
- HDF5 has no pointer to a metadata region. libhdf5's paged aggregation
(`H5Pset_file_space_strategy(H5F_FSPACE_STRATEGY_PAGE)`,
`h5repack -S PAGE -G <size>`) keeps metadata and raw data in separate pages,
and "setting an appropriate page size can have all internal file metadata
in just one page" ([Cloud-Optimized HDF/NetCDF guide][cog]) — but only
when the writer chose it: 4 of 611 corpus files (§2).
- Without paging, metadata is scattered: in B the chunk index is spread over
14 MiB-blocks of a 48 MB file. Prefetching the first N MiB (what earthaccess'
block cache effectively does for "shared metadata at the file beginning"
([earthaccess][ea])) is a good *heuristic*, not a correctness basis.
- A sparse buffer still presents as a `&[u8]` of the whole file's length, so
every metadata read outside the prefetched part must be caught — back to (b).
Useful as a **policy** on top of (a) (prefetch the first block, use the page
size of paged files as the block size), not as the architecture.
### (d) mmap a userfaultfd- or FUSE-backed file
Keep `&[u8]` everywhere and let the kernel fault in remote pages
(userfaultfd handler, or a FUSE file system doing range GETs).
- Linux-only (userfaultfd) or needs a FUSE mount and privileges; nothing on
macOS/Windows without a kernel extension; impossible in wasm.
- Faults are synchronous and uninterruptible from the parser's view: a network
error becomes `SIGBUS`, a stalled request hangs the thread.
- No batching: raw data arrives one fault (4 KiB, or the FUSE read size) at a
time unless the handler guesses read-ahead.
- 32-bit address space is still exhausted by mapping a large file.
Rejected. It is how one would retrofit a C library that cannot change; we can.
### How other readers do it
- **libhdf5 `ros3`** replaces each POSIX read with an S3 range GET
([h5py docs][h5py-file], [HDF Group, cloud storage options][hdfg-cloud]).
libhdf5 has a real metadata cache above the driver, and HDF5 2.2.0 added an
I/O block cache to ros3 "to reduce the number of requests to S3 for files
not using paged allocation", while paged files use the page buffer
([HDF5 2.2.0 release][hdf5-220], [ros3 issue #4700][ros3-4700]). The page
buffer (`page_buf_size`) is "only allowed for HDF5 files created with
fs_strategy='page'" ([h5py docs][h5py-file]). PyPI h5py wheels do not
include ros3.
- **h5py + fsspec** hands libhdf5 a Python file object; each libhdf5 read
becomes a `read()` on an fsspec file, whose cache decides the requests.
fsspec's old default `readahead` cache requested 16x more data than
`blockcache` when opening HDF5 files; earthaccess now defaults to
`blockcache` with 4–16 MiB blocks by file size ([earthaccess][ea]); the
Cloud-Optimized guide recommends `blockcache` plus h5py's `page_buf_size`
and `rdcc_nbytes` ([guide][cog]). Our §2 comparison uses exactly this
path, without fsspec's cache.
- **pyfive** (pure Python, like us) supports lazy loading "on both Posix and
S3 filesystems" through fsspec, reads a variable's attributes and chunk
B-tree when it is accessed, and can merge chunk range requests with
fsspec's `merge_range_requests` ([pyfive][pyfive], [pyfive #257][pyfive-257]).
- **h5wasm** (libhdf5 compiled with Emscripten) backs a file with
`FS.createLazyFile`, which "can use range requests to incrementally access
the h5 file over the wire" ([h5wasm][h5wasm]); Emscripten's lazy files use
synchronous XHR, which browsers only allow in Web Workers
([Emscripten FS API][emfs]). Files larger than memory remain an open issue
([h5wasm #40][h5wasm-40]).
- **jsfive** (pure JS port of pyfive) reads from an `ArrayBuffer` of the whole
file ([jsfive][jsfive]); no lazy loading.
- **Rust `object_store`** (0.14): one `ObjectStore` trait over S3, GCS, Azure,
HTTP/WebDAV, local files and memory; `get_ranges` "will automatically
coalesce adjacent ranges into an appropriate number of parallel requests",
with `OBJECT_STORE_COALESCE_DEFAULT` as the gap below which ranges merge
([docs.rs][os]). It builds for wasm32 except the local-filesystem and
some chunked-upload parts.
## 4. Recommendation and migration plan
Adopt **(a)**, with a block cache as a required part of every non-local
backend, **(c)** as a cache policy, and the wasm path through the restartable
`NeedBytes` mode. Every milestone keeps `main` green: `cargo test
--workspace`, clippy, the conformance gate at 575/697 unchanged, and the mmap
fast path within benchmark noise.
**M0 — prerequisites (≈1 week).**
- Dense-group name lookup through the v2 B-tree name index (Jenkins hash,
record type 5), instead of enumerating all links; same for creation-order
and attribute name lookups. §2 shows 936 ranges → a handful. Regression
test: count links decoded for one lookup in a 35 001-link group.
- One checked helper for address → index conversion; replace the 133
`as usize` casts. On 32-bit an address past `usize::MAX` is a clean error
(today it truncates or panics).
- `Group::dataset(name)`/`Group::group(name)` call `children()`, which
re-reads the group's whole link list on every lookup, so walking a group of
n children decodes its links O(n) times. Look names up through the index
(above) and let a listing hand out its entries, so the cache has less to
absorb.
**M1 — metadata over the trait, in-memory impl identical to today (2–3 weeks).**
- Add `Storage` (above) to `clawhdf5-format`, `no_std`-compatible, with
`impl Storage for [u8]`. Add a storage error variant — `FormatError` and the
facade `Error` are not `#[non_exhaustive]`, so this is a breaking change for
exhaustive matches: bundle it with the next major version, or mark both
enums `#[non_exhaustive]` first.
- Convert modules bottom-up — superblock, object header, local/global heap,
B-tree v1/v2, fractal heap, fixed/extensible array, symbol table, group
v1/v2, shared messages, attributes, fill value, data layout — one commit
each. The old `&[u8]` signature stays as a thin wrapper over the new one
(`fn parse(data: &[u8], ..) { parse_in(data as &dyn Storage, ..) }`), so
callers and the other crates don't move yet.
- Replace the 5 open-ended slices and 38 `len()` checks with bounded reads.
**M2 — raw data over the trait (1–2 weeks).**
- `data_read`, `chunked_read`, `parallel_read`, `partial_read`, `vds`,
VL/global heap. Chunked reads first collect the chunk extents the selection
needs, then call `read_ranges` once and decompress in parallel as today.
- Zero-copy stays: when `as_contiguous()` is `Some`, contiguous reads return
borrowed slices; `read_raw_ref`/`read_*_zerocopy`/`File::as_bytes` keep
their signatures and return a clear "not available for this storage" error
on other backends (they already return `Option`/`Result`).
- Facade: `File::open_storage(Box<dyn Storage + Send + Sync>)`; `File::open`
keeps mmap and `from_bytes` keeps `Vec`, both through `impl Storage for [u8]`.
**M3 — HTTP/S3 backend (1–2 weeks).**
- `clawhdf5-io`, feature `remote` (off by default, so the default tree stays
free of C and TLS stacks): `RangeStorage` over `object_store` (HTTP, S3, GCS,
Azure), with a `BlockCache` (LRU, block size configurable, default 1 MiB per
§2; the page size for paged files; the first block prefetched on open) and
a request counter exposed for tests and users.
- Python bindings: `clawhdf5.File("s3://…")` / `https://` through it.
**M4 — wasm lazy loading (1–2 weeks).**
- `clawhdf5-wasm`: `openUrl(url) -> Promise<H5File>` backed by `fetch` with a
`Range` header, on the main thread, via the restartable `NeedBytes` loop (no
Worker, no synchronous XHR — the thing h5wasm's lazy files need). Falls back
to a whole download when the server does not answer 206.
- `examples/wasm-viewer`: open by URL.
**M5 — SWMR and growth (later, separate design).** `Storage::len()` may grow;
add `File::refresh()` that re-reads the superblock/EOF and invalidates cached
blocks past the old end. Needs libhdf5 SWMR semantics research first.
Total: roughly 6–10 engineer-weeks for M0–M4 (estimate, not measured).
### Keeping the local fast path
- `impl Storage for [u8]` returns `Cow::Borrowed` — no copy, no allocation.
- Hot loops (raw-data copies, contiguous typed reads, `read_selection_native`)
branch once on `as_contiguous()` and then run today's code.
- `&dyn` dispatch is per structure, not per byte; parse code keeps working on
the returned slice.
- Gate: `crates/clawhdf5/benches/mmap_bench.rs`, the concurrent-read benches in
`clawhdf5-bench`, and the conformance run time, before and after each M1/M2
commit, on an otherwise idle machine. Anything outside noise blocks the
commit.
### Risks
- **Silent regressions on local files** — mitigated by the bench gate above
and by M1 being a pure refactor (every conformance hash identical).
- **Two reads per structure** (prefix, then body) could double requests on a
cold cache. Block-aligned caching makes the second read a hit; the measured
block counts already include this pattern.
- **API break** — new error variant (see M1); `as_bytes()`-style APIs become
fallible for non-local storage. ClawBrainHub uses `File`, `FileBuilder`,
`AttrValue`, `Selection` on local files only, so it is unaffected by
behaviour, only by exhaustive matches on `Error`.
- **Restartable parsing** assumes operations are pure over the storage. The
chunk cache and metadata cache must only be filled by completed reads.
- **Cache memory** — the block cache needs a byte budget and eviction, which
(unlike option (b)) is sound because parsers hold `Cow`s, not borrows into
the cache.
### Testing
- **In-memory equivalence:** a `CountingStorage` wrapper over `[u8]` that
records every `read_at`; run the conformance probe through it and require
identical results. It also produces the request counts of §2 without the
trap-flag tracer.
- **Adversarial storage:** a wrapper that returns short reads, errors on the
Nth request, or serves blocks of 1 byte, to prove every miss is an `Err`
and never wrong data.
- **HTTP:** an in-process server on `127.0.0.1` (std `TcpListener`) that
honours `Range`, answers 206/416, can refuse ranges (200), and counts
requests. Tests assert both correctness against h5py and **request budgets**
(e.g. listing file A with a 1 MiB block cache takes ≤ 3 requests) so a
change that makes the reader chattier fails CI.
- **S3:** `object_store`'s in-memory store in unit tests; a MinIO or real
bucket only in an opt-in job.
- **wasm:** the existing Node/Chromium harness in `examples/wasm-viewer/test`
with a local range-capable server.
- **32-bit:** a `wasm32` or `i686` build that opens a sparse > 4 GiB file
through the counting storage.
## Tools
- `docs/design/tools/inventory.py` — §1 tables (`--list`, `--patterns`).
- `docs/design/tools/range-trace/` — §2 tracer (standalone crate, x86-64
Linux, not part of the workspace).
- `docs/design/tools/libhdf5_reads.py` — §2 libhdf5 comparison and chunk
extents.
[cog]: https://guide.cloudnativegeo.org/cloud-optimized-netcdf4-hdf5/
[ea]: https://earthaccess.readthedocs.io/en/latest/user/explanation/fsspec/
[h5py-file]: https://docs.h5py.org/en/stable/high/file.html
[hdfg-cloud]: https://www.hdfgroup.org/2022/08/08/cloud-storage-options-for-hdf5/
[hdf5-220]: https://www.hdfgroup.org/2026/07/30/release-of-hdf5-2-2-0-and-two-august-events-newsletter-210/
[ros3-4700]: https://github.com/HDFGroup/hdf5/issues/4700
[pyfive]: https://pyfive.readthedocs.io/en/latest/quickstart/usage.html
[pyfive-257]: https://github.com/NCAS-CMS/pyfive/issues/257
[h5wasm]: https://github.com/usnistgov/h5wasm
[h5wasm-40]: https://github.com/usnistgov/h5wasm/issues/40
[emfs]: https://emscripten.org/docs/api_reference/Filesystem-API.html
[jsfive]: https://github.com/usnistgov/jsfive
[os]: https://docs.rs/object_store/latest/object_store/
docs/design/tools/inventory.py
+131
View File
@@ -0,0 +1,131 @@
#!/usr/bin/env python3
"""Inventory of whole-file `&[u8]` parameters in the clawhdf5 workspace.
Used by docs/design/range-reads.md. Run from the repository root:
python3 docs/design/tools/inventory.py # per-file table
python3 docs/design/tools/inventory.py --list # every signature
python3 docs/design/tools/inventory.py --patterns # read patterns per crate
A function counts as taking "the whole file" when it has a parameter named
`file_data: &[u8]` (the repository convention), or a `data` / `file` / `buf` /
`bytes` / `mmap` parameter of type `&[u8]` *together with* a parameter whose
name says it is a file address (`*address*`, `*addr*`, `*offset*` of an
integer type). The second rule is a heuristic; `--list` prints every match so
it can be checked by eye. Code after the first `#[cfg(test)] mod ... {` in a
file is excluded (the repository keeps unit tests at the end of each file),
as are the tests/ and benches/ directories.
"""
import os
import re
import sys
from collections import defaultdict
ROOT = os.getcwd()
FN_RE = re.compile(r"\bfn\s+([A-Za-z_][A-Za-z0-9_]*)\s*(<[^()]*?>)?\s*\(", re.S)
PARAM_RE = re.compile(r"([A-Za-z_][A-Za-z0-9_]*)\s*:\s*&(?:'[a-z_]+\s+)?\[u8\]")
ADDR_RE = re.compile(r"\b([a-z_]*(?:address|addr|offset)[a-z_]*)\s*:\s*(?:u64|usize|u32)")
WHOLE_NAMES = {"data", "file", "buf", "bytes", "file_bytes", "mmap"}
def signature(src, start):
depth, i = 0, start
while i < len(src):
c = src[i]
if c == "(":
depth += 1
elif c == ")":
depth -= 1
if depth == 0:
return src[start + 1 : i]
i += 1
return ""
def non_test_source(src):
m = re.search(r"#\[cfg\(test\)\]\s*mod\s+\w+\s*\{", src)
return src[: m.start()] if m else src
PATTERNS = [
("`file_data` passed on (call sites)", re.compile(r"[(,]\s*&?file_data\s*[,)]")),
("`file_data[..]` slicing", re.compile(r"\bfile_data\s*\[")),
("open-ended `file_data[x..]`", re.compile(r"\bfile_data\s*\[[^\]]*\.\.\s*\]")),
("`file_data.get(..)`", re.compile(r"\bfile_data\s*\.\s*get\s*\(")),
("`file_data.len()`", re.compile(r"\bfile_data\s*\.\s*len\s*\(\)")),
("address/offset `as usize` casts", re.compile(r"\b[a-z_]*(?:addr|address|offset)[a-z_]*\s+as\s+usize")),
("`ObjectHeader::parse(` calls", re.compile(r"ObjectHeader::parse\s*\(")),
("`.as_bytes()` on a file/reader", re.compile(r"\b(?:file|reader|data|self\.file|self\.data|self\.file\.data|root\.file)\s*\.\s*as_bytes\s*\(\)")),
]
def patterns():
"""Per-crate counts of the read patterns (non-test code only)."""
per_crate = defaultdict(lambda: [0] * len(PATTERNS))
for crate in sorted(os.listdir(os.path.join(ROOT, "crates"))):
srcdir = os.path.join(ROOT, "crates", crate, "src")
for dp, _, fns in os.walk(srcdir):
for fn in fns:
if not fn.endswith(".rs"):
continue
with open(os.path.join(dp, fn), encoding="utf-8") as fh:
src = non_test_source(fh.read())
for i, (_, rx) in enumerate(PATTERNS):
per_crate[crate][i] += len(rx.findall(src))
print("| crate | " + " | ".join(n for n, _ in PATTERNS) + " |")
print("|---|" + "---:|" * len(PATTERNS))
tot = [0] * len(PATTERNS)
for c, v in sorted(per_crate.items()):
if any(v):
print("| %s | %s |" % (c, " | ".join(str(x) for x in v)))
tot = [a + b for a, b in zip(tot, v)]
print("| **total** | %s |" % " | ".join("**%d**" % x for x in tot))
def main():
if "--patterns" in sys.argv:
patterns()
return
per_file = defaultdict(lambda: [0, 0, 0]) # [file_data, heuristic, pub]
rows = []
for crate in sorted(os.listdir(os.path.join(ROOT, "crates"))):
srcdir = os.path.join(ROOT, "crates", crate, "src")
for dp, _, fns in os.walk(srcdir):
for fn in sorted(fns):
if not fn.endswith(".rs"):
continue
path = os.path.join(dp, fn)
with open(path, encoding="utf-8") as fh:
src = non_test_source(fh.read())
for m in FN_RE.finditer(src):
sig = signature(src, m.end() - 1)
params = PARAM_RE.findall(sig)
kind = None
if "file_data" in params:
kind = "file_data"
elif any(p in WHOLE_NAMES for p in params) and ADDR_RE.search(sig):
kind = "heuristic"
if not kind:
continue
rel = os.path.relpath(path, ROOT)
line_start = src.rfind("\n", 0, m.start()) + 1
is_pub = src[line_start : m.start()].strip().startswith("pub")
per_file[rel][0 if kind == "file_data" else 1] += 1
per_file[rel][2] += int(is_pub)
line = src.count("\n", 0, m.start()) + 1
rows.append((rel, line, m.group(1), kind, is_pub))
if "--list" in sys.argv:
for r in rows:
print("%s:%d %s [%s%s]" % (r[0], r[1], r[2], r[3], ", pub" if r[4] else ""))
return
print("| file | `file_data` fns | other whole-slice fns (heuristic) | of which `pub` |")
print("|---|---:|---:|---:|")
tot = [0, 0, 0]
for f, (a, b, p) in sorted(per_file.items(), key=lambda kv: (-(kv[1][0] + kv[1][1]), kv[0])):
print("| %s | %d | %d | %d |" % (f, a, b, p))
tot = [tot[0] + a, tot[1] + b, tot[2] + p]
print("| **total** | **%d** | **%d** | **%d** |" % tuple(tot))
if __name__ == "__main__":
main()
docs/design/tools/libhdf5_reads.py
+98
View File
@@ -0,0 +1,98 @@
#!/usr/bin/env python3
"""The libhdf5 side of the range-read measurement (docs/design/range-reads.md).
libhdf5_reads.py FILE DATASET # count libhdf5's reads
libhdf5_reads.py FILE DATASET --extents # print DATASET's stored extents
Counting: the file is opened through h5py's `fileobj` driver with a Python
file-like object that logs every `readinto`/`read` libhdf5 makes (this is how
h5py + fsspec reads remote files: each call becomes a range request unless
fsspec's own block cache absorbs it). The phases mirror range-trace: open,
list (visit every object; shape and dtype of every dataset), read (the whole
dataset). Default h5py settings (libhdf5's metadata cache, 64 KiB sieve
buffer, 1 MiB raw chunk cache) apply.
--extents prints `offset size` lines for the dataset's stored data (the
contiguous block, or every allocated chunk), the input range-trace uses to
split its read phase into metadata and raw data.
"""
import sys
import h5py
class LoggingFile:
def __init__(self, path):
self.f = open(path, "rb")
self.pos = 0
self.log = []
def seek(self, off, whence=0):
self.pos = self.f.seek(off, whence)
return self.pos
def tell(self):
return self.pos
def readinto(self, b):
n = self.f.readinto(b)
self.log.append((self.pos, n))
self.pos += n
return n
def read(self, size=-1):
data = self.f.read(size)
self.log.append((self.pos, len(data)))
self.pos += len(data)
return data
def extents(path, name):
with h5py.File(path, "r") as f:
ds = f[name]
if ds.chunks is None:
off = ds.id.get_offset()
if off is not None:
print(off, ds.id.get_storage_size())
return
for i in range(ds.id.get_num_chunks()):
info = ds.id.get_chunk_info(i)
print(info.byte_offset, info.size)
def summarise(label, log):
n = len(log)
total = sum(s for _, s in log)
distinct = len(set(log))
print("| %s | %d | %d | %d |" % (label, n, distinct, total))
def count(path, name):
lf = LoggingFile(path)
f = h5py.File(lf, "r")
opened = list(lf.log)
lf.log.clear()
def visit(_n, obj):
if isinstance(obj, h5py.Dataset):
obj.shape, obj.dtype
f.visititems(visit)
listed = list(lf.log)
lf.log.clear()
f[name][()]
readlog = list(lf.log)
f.close()
print("| phase | read calls | distinct (offset, len) | bytes |")
print("|---|---:|---:|---:|")
summarise("open", opened)
summarise("list", listed)
summarise("read", readlog)
summarise("open+list+read", opened + listed + readlog)
if __name__ == "__main__":
if len(sys.argv) >= 4 and sys.argv[3] == "--extents":
extents(sys.argv[1], sys.argv[2])
else:
count(sys.argv[1], sys.argv[2])
docs/design/tools/range-trace/Cargo.toml
+17
View File
@@ -0,0 +1,17 @@
[package]
name = "range-trace"
version = "0.1.0"
edition = "2024"
rust-version = "1.92"
publish = false
description = "Records which byte ranges of an HDF5 file the clawhdf5 facade reads (docs/design/range-reads.md)"
# Outside the main workspace on purpose: a measurement tool for a design
# document, never built by `cargo test --workspace`. x86-64 Linux only.
[workspace]
[dependencies]
# Default features minus nothing: `parallel` is off by default, and the tracer
# relies on every read happening on the main thread.
clawhdf5 = { path = "../../../../crates/clawhdf5" }
libc = "0.2"
docs/design/tools/range-trace/src/main.rs
+375
View File
@@ -0,0 +1,375 @@
//! range-trace: which bytes of an HDF5 file does clawhdf5 read?
//!
//! Measurement tool for `docs/design/range-reads.md` (x86-64 Linux only).
//!
//! The file is loaded into a page-aligned buffer and handed to
//! `clawhdf5::File::from_bytes`, so the library parses it exactly as it does
//! today. The buffer is then `mprotect`ed to `PROT_NONE`. Every load from it
//! faults; the SIGSEGV handler logs the exact faulting address, makes that
//! page readable and sets the x86 trap flag, so the CPU single-steps the one
//! instruction and the SIGTRAP handler re-protects the page. Every load
//! instruction that touches the file is therefore recorded (with its first
//! byte; widths are not decoded, see `ACCESS_WIDTH`).
//!
//! Bulk copies (raw data) would fault once per load; a page that faults more
//! than `BULK_THRESHOLD` times in one phase is left readable for the rest of
//! that phase and counted as wholly read ("bulk page").
//!
//! Phases: `open` (superblock), `list` (walk every group; for every dataset
//! its shape and dtype, i.e. what `h5ls -r -v` or a tree view needs), and
//! `read` (read the named dataset in full through `read_selection(All)`).
//!
//! Usage: range-trace FILE DATASET_PATH [RAW_EXTENTS]
//!
//! RAW_EXTENTS (optional) lists the dataset's stored data as `offset size`
//! lines (absolute file offsets; `libhdf5_reads.py --extents` writes it from
//! h5py). With it the `read` phase is split into `read (metadata)`, the
//! loads outside those extents, and `read (raw data)`, the loads inside.
//!
//! Every read must happen on this thread: build without the facade's
//! `parallel` feature (off by default).
use std::alloc::{Layout, alloc_zeroed};
use std::collections::BTreeSet;
use std::sync::atomic::{AtomicPtr, AtomicUsize, Ordering};
use clawhdf5::{File, Group, Selection};
const PAGE: usize = 4096;
const BULK_THRESHOLD: u32 = 4096;
/// Bytes assumed read by one logged load (the widest scalar load the parsers
/// issue; SIMD copies are bulk anyway).
const ACCESS_WIDTH: u64 = 8;
/// Two accesses closer than this belong to the same structure, i.e. would be
/// one range request.
const MERGE_GAP: u64 = 64;
const LOG_CAP: usize = 64 << 20;
static BUF_START: AtomicUsize = AtomicUsize::new(0);
static BUF_LEN: AtomicUsize = AtomicUsize::new(0);
static LOG: AtomicPtr<u64> = AtomicPtr::new(std::ptr::null_mut());
static LOG_LEN: AtomicUsize = AtomicUsize::new(0);
static HITS: AtomicPtr<u32> = AtomicPtr::new(std::ptr::null_mut());
static BULK: AtomicPtr<u8> = AtomicPtr::new(std::ptr::null_mut());
static PENDING: [AtomicUsize; 8] = [const { AtomicUsize::new(0) }; 8];
static PENDING_N: AtomicUsize = AtomicUsize::new(0);
const TF: i64 = 0x100;
extern "C" fn on_segv(_sig: libc::c_int, info: *mut libc::siginfo_t, ctx: *mut libc::c_void) {
unsafe {
let addr = (*info).si_addr() as usize;
let start = BUF_START.load(Ordering::Relaxed);
let len = BUF_LEN.load(Ordering::Relaxed);
if addr < start || addr >= start + len {
// A genuine crash: restore the default action and re-fault.
libc::signal(libc::SIGSEGV, libc::SIG_DFL);
return;
}
let off = addr - start;
let n = LOG_LEN.load(Ordering::Relaxed);
if n < LOG_CAP {
*LOG.load(Ordering::Relaxed).add(n) = off as u64;
LOG_LEN.store(n + 1, Ordering::Relaxed);
}
let page = off / PAGE;
let hits = HITS.load(Ordering::Relaxed).add(page);
*hits += 1;
libc::mprotect(
(start + page * PAGE) as *mut libc::c_void,
PAGE,
libc::PROT_READ | libc::PROT_WRITE,
);
if *hits >= BULK_THRESHOLD {
*BULK.load(Ordering::Relaxed).add(page) = 1;
} else {
let p = PENDING_N.load(Ordering::Relaxed);
if p < PENDING.len() {
PENDING[p].store(page, Ordering::Relaxed);
PENDING_N.store(p + 1, Ordering::Relaxed);
}
}
let uc = ctx as *mut libc::ucontext_t;
(*uc).uc_mcontext.gregs[libc::REG_EFL as usize] |= TF;
}
}
extern "C" fn on_trap(_sig: libc::c_int, _info: *mut libc::siginfo_t, ctx: *mut libc::c_void) {
unsafe {
let start = BUF_START.load(Ordering::Relaxed);
let n = PENDING_N.load(Ordering::Relaxed);
for p in PENDING.iter().take(n) {
let page = p.load(Ordering::Relaxed);
libc::mprotect(
(start + page * PAGE) as *mut libc::c_void,
PAGE,
libc::PROT_NONE,
);
}
PENDING_N.store(0, Ordering::Relaxed);
let uc = ctx as *mut libc::ucontext_t;
(*uc).uc_mcontext.gregs[libc::REG_EFL as usize] &= !TF;
}
}
fn install(
sig: libc::c_int,
h: extern "C" fn(libc::c_int, *mut libc::siginfo_t, *mut libc::c_void),
) {
unsafe {
let mut sa: libc::sigaction = std::mem::zeroed();
sa.sa_sigaction = h as usize;
sa.sa_flags = libc::SA_SIGINFO | libc::SA_NODEFER;
libc::sigemptyset(&mut sa.sa_mask);
assert_eq!(libc::sigaction(sig, &sa, std::ptr::null_mut()), 0);
}
}
fn protect(prot: libc::c_int) {
let start = BUF_START.load(Ordering::Relaxed);
let len = BUF_LEN.load(Ordering::Relaxed);
unsafe {
assert_eq!(libc::mprotect(start as *mut libc::c_void, len, prot), 0);
}
}
struct Phase {
name: &'static str,
log: Vec<u64>,
bulk_pages: Vec<u64>,
}
/// Start a phase: clear the per-page state and protect the buffer.
fn begin() {
let pages = BUF_LEN.load(Ordering::Relaxed) / PAGE;
unsafe {
std::ptr::write_bytes(HITS.load(Ordering::Relaxed), 0, pages);
std::ptr::write_bytes(BULK.load(Ordering::Relaxed), 0, pages);
}
LOG_LEN.store(0, Ordering::Relaxed);
protect(libc::PROT_NONE);
}
fn end(name: &'static str) -> Phase {
protect(libc::PROT_READ | libc::PROT_WRITE);
let n = LOG_LEN.load(Ordering::Relaxed);
let log = unsafe { std::slice::from_raw_parts(LOG.load(Ordering::Relaxed), n) }.to_vec();
let pages = BUF_LEN.load(Ordering::Relaxed) / PAGE;
let bulk = unsafe { std::slice::from_raw_parts(BULK.load(Ordering::Relaxed), pages) };
let bulk_pages = (0..pages)
.filter(|&p| bulk[p] != 0)
.map(|p| p as u64)
.collect();
if n == LOG_CAP {
eprintln!("warning: access log full in phase {name}");
}
Phase {
name,
log,
bulk_pages,
}
}
/// Byte intervals [lo, hi) read in a phase, merged when closer than `gap`.
fn ranges(ph: &[&Phase], gap: u64) -> Vec<(u64, u64)> {
let mut iv: Vec<(u64, u64)> = Vec::new();
for p in ph {
iv.extend(p.log.iter().map(|&o| (o, o + ACCESS_WIDTH)));
iv.extend(
p.bulk_pages
.iter()
.map(|&pg| (pg * PAGE as u64, (pg + 1) * PAGE as u64)),
);
}
iv.sort_unstable();
let mut out: Vec<(u64, u64)> = Vec::new();
for (lo, hi) in iv {
match out.last_mut() {
Some(last) if lo <= last.1 + gap => last.1 = last.1.max(hi),
_ => out.push((lo, hi)),
}
}
out
}
/// Requests a reader with no cache at all would make: a new request each
/// time the access stream leaves the neighbourhood of the current run.
fn uncached_requests(p: &Phase) -> usize {
let mut n = 0;
let (mut lo, mut hi) = (u64::MAX, 0u64);
for &o in &p.log {
if lo != u64::MAX && o + MERGE_GAP >= lo && o <= hi + MERGE_GAP {
lo = lo.min(o);
hi = hi.max(o + ACCESS_WIDTH);
} else {
n += 1;
lo = o;
hi = o + ACCESS_WIDTH;
}
}
n + p.bulk_pages.len()
}
fn blocks(ph: &[&Phase], block: u64) -> usize {
let mut set = BTreeSet::new();
for (lo, hi) in ranges(ph, 0) {
for b in lo / block..=(hi - 1) / block {
set.insert(b);
}
}
set.len()
}
fn read_extents(path: &str) -> Vec<(u64, u64)> {
let text = std::fs::read_to_string(path).expect("read extents");
let mut v: Vec<(u64, u64)> = text
.lines()
.filter_map(|l| {
let mut it = l.split_whitespace().map(|t| t.parse::<u64>());
match (it.next(), it.next()) {
(Some(Ok(o)), Some(Ok(n))) => Some((o, o + n)),
_ => None,
}
})
.collect();
v.sort_unstable();
v
}
fn in_extents(ext: &[(u64, u64)], lo: u64, hi: u64) -> bool {
let i = ext.partition_point(|e| e.1 <= lo);
i < ext.len() && ext[i].0 < hi
}
/// Split a phase into loads outside and inside the raw-data extents. A bulk
/// page counts as raw data when it overlaps an extent.
fn split_raw(p: &Phase, ext: &[(u64, u64)]) -> (Phase, Phase) {
let (mut m, mut r) = (Vec::new(), Vec::new());
for &o in &p.log {
if in_extents(ext, o, o + 1) {
r.push(o)
} else {
m.push(o)
}
}
let pg = PAGE as u64;
let (bm, br): (Vec<u64>, Vec<u64>) = p
.bulk_pages
.iter()
.partition(|&&b| !in_extents(ext, b * pg, (b + 1) * pg));
(
Phase {
name: "read (metadata)",
log: m,
bulk_pages: bm,
},
Phase {
name: "read (raw data)",
log: r,
bulk_pages: br,
},
)
}
fn walk(g: &Group<'_>, path: &str, objs: &mut usize) {
for name in g.datasets().unwrap_or_default() {
*objs += 1;
if let Ok(ds) = g.dataset(&name) {
let _ = ds.shape();
let _ = ds.dtype();
}
}
for name in g.groups().unwrap_or_default() {
*objs += 1;
if let Ok(sub) = g.group(&name) {
walk(&sub, &format!("{path}/{name}"), objs);
}
}
}
fn main() {
let args: Vec<String> = std::env::args().collect();
if args.len() != 3 && args.len() != 4 {
eprintln!("usage: range-trace FILE DATASET_PATH [RAW_EXTENTS]");
std::process::exit(2);
}
let bytes = std::fs::read(&args[1]).expect("read file");
let len = bytes.len();
let cap = len.div_ceil(PAGE).max(1) * PAGE;
let pages = cap / PAGE;
// Page-aligned buffer so that protection covers exactly the file. The
// Vec is never dropped (its layout differs from Vec's own), see the end.
let ptr = unsafe { alloc_zeroed(Layout::from_size_align(cap, PAGE).unwrap()) };
unsafe { std::ptr::copy_nonoverlapping(bytes.as_ptr(), ptr, len) };
drop(bytes);
let buf = unsafe { Vec::from_raw_parts(ptr, len, cap) };
BUF_START.store(ptr as usize, Ordering::Relaxed);
BUF_LEN.store(cap, Ordering::Relaxed);
let mut log = vec![0u64; LOG_CAP];
LOG.store(log.as_mut_ptr(), Ordering::Relaxed);
let mut hits = vec![0u32; pages];
HITS.store(hits.as_mut_ptr(), Ordering::Relaxed);
let mut bulk = vec![0u8; pages];
BULK.store(bulk.as_mut_ptr(), Ordering::Relaxed);
install(libc::SIGSEGV, on_segv);
install(libc::SIGTRAP, on_trap);
begin();
let file = File::from_bytes(buf).expect("open");
let p_open = end("open");
begin();
let mut objs = 0;
walk(&file.root(), "", &mut objs);
let p_list = end("list");
begin();
let out = file
.dataset(&args[2])
.and_then(|d| d.read_selection(&Selection::All))
.expect("read dataset");
let p_read = end("read");
println!("file: {} ({} bytes), objects listed: {objs}", args[1], len);
println!("dataset: {} ({} bytes decoded)", args[2], out.len());
println!(
"| phase | loads logged | bulk 4K pages | uncached requests | distinct ranges (gap<{MERGE_GAP}B) | bytes in ranges | 4 KiB blocks | 64 KiB blocks | 1 MiB blocks |"
);
println!("|---|---:|---:|---:|---:|---:|---:|---:|---:|");
let row = |label: String, ph: &[&Phase], uncached: usize| {
let r = ranges(ph, MERGE_GAP);
let bytes: u64 = r.iter().map(|(a, b)| b - a).sum();
let loads: usize = ph.iter().map(|p| p.log.len()).sum();
let bulk: usize = ph.iter().map(|p| p.bulk_pages.len()).sum();
println!(
"| {label} | {loads} | {bulk} | {uncached} | {} | {bytes} | {} | {} | {} |",
r.len(),
blocks(ph, 4 << 10),
blocks(ph, 64 << 10),
blocks(ph, 1 << 20)
);
};
for p in [&p_open, &p_list, &p_read] {
row(p.name.to_string(), &[p], uncached_requests(p));
}
if let Some(path) = args.get(3) {
let (meta, raw) = split_raw(&p_read, &read_extents(path));
for p in [&meta, &raw] {
row(p.name.to_string(), &[p], uncached_requests(p));
}
let all = [&p_open, &p_list, &meta];
let unc: usize = all.iter().map(|p| uncached_requests(p)).sum();
row("all metadata".into(), &all, unc);
}
let all = [&p_open, &p_list, &p_read];
let unc: usize = all.iter().map(|p| uncached_requests(p)).sum();
row("open+list+read".into(), &all, unc);
let meta = [&p_open, &p_list];
let unc: usize = meta.iter().map(|p| uncached_requests(p)).sum();
row("open+list".into(), &meta, unc);
// The File owns a buffer whose layout Vec does not know; never drop it.
std::mem::forget(file);
std::mem::forget(log);
std::mem::forget(hits);
std::mem::forget(bulk);
}