Merge branch 'feat/p3-m3-remote' into feat/p3-remote-editor
# Conflicts: # crates/clawhdf5/tests/storage_equivalence.rs
This commit is contained in:
@@ -5,8 +5,10 @@ change. Progress: M0 and M1 are done, and so is M2 (branch
|
||||
`feat/p3-m2-raw-data`): every read path of the format crate works through
|
||||
`Storage`, v2 B-trees, dense groups and raw data included, and
|
||||
`File::open_storage` gives the facade's read API over any `Storage` (see
|
||||
`CHANGELOG.md`, "Range reads, milestone M2"). M3 (a remote backend with
|
||||
its block cache) is next. Every count in §1–§2 was
|
||||
`CHANGELOG.md`, "Range reads, milestone M2"). M3 is done on branch
|
||||
`feat/p3-m3-remote`: the `clawhdf5-remote` crate (block cache, HTTP(S),
|
||||
object stores) and URLs in `h5rs` (see the M3 status below). M4 (wasm) is
|
||||
next. Every count in §1–§2 was
|
||||
taken on `tank` on 2026-09-26 at commit `de2a53f`, with the commands given
|
||||
next to it. No timing numbers appear here on purpose: the machine was shared
|
||||
with other build jobs when this was written.
|
||||
@@ -462,6 +464,44 @@ fast path within benchmark noise.
|
||||
§2; the page size for paged files; the first block prefetched on open) and
|
||||
a request counter exposed for tests and users.
|
||||
- Python bindings: `clawhdf5.File("s3://…")` / `https://` through it.
|
||||
- *Status 2026-09-26:* done on branch `feat/p3-m3-remote`, except the
|
||||
Python bindings, with these choices:
|
||||
- A new crate, `clawhdf5-remote`, instead of a `remote` feature of
|
||||
`clawhdf5-io`: `open_url` returns a `clawhdf5::File`, and `clawhdf5-io`
|
||||
sits below the facade.
|
||||
- HTTP(S) through `ureq` (`HttpStorage`), not object_store's HTTP store:
|
||||
that one pulls reqwest with aws-lc-rs (C), while plain HTTP through
|
||||
ureq builds no C, so it is the default feature; `https` adds rustls
|
||||
with ring. S3/GCS/Azure go through `object_store` (`ObjectStoreStorage`,
|
||||
features `s3`/`gcs`/`azure`, opt-in because of aws-lc-rs); the
|
||||
`object-store` feature alone (in-memory, local files, a store you
|
||||
build) is pure Rust. object_store is async: each read runs on a
|
||||
two-thread tokio runtime of the storage's own while the caller waits,
|
||||
so the caller's context (a plain thread, `spawn_blocking`, another
|
||||
runtime) does not matter.
|
||||
- `BlockCache` (any `Storage`): 1 MiB blocks and a 64 MiB LRU budget by
|
||||
default, the first block fetched at open (for HTTP by the request
|
||||
that learns the length), the missing blocks of one read fetched as
|
||||
runs of consecutive blocks in one parallel batch, per-block in-flight
|
||||
deduplication across threads, and reads that miss more than half the
|
||||
budget not kept. The page size of paged files is not used as the block
|
||||
size yet. `CacheStats` and `HttpStats` count requests and bytes.
|
||||
- The file is pinned at open by ETag (else Last-Modified, or the object's
|
||||
version) and length; a change is an error, not mixed data.
|
||||
- `h5rs` (feature `remote`) reads through the new `File::storage()`, the
|
||||
file's view as a `Storage`, so its parsing works on remote files; its
|
||||
`check` downloads the file whole.
|
||||
- Measured with `crates/clawhdf5-remote/tests/http.rs` (tank, 2026-09-26,
|
||||
`CLAWHDF5_REMOTE_CORPUS=conformance/.cache/corpus
|
||||
CLAWHDF5_REMOTE_REPORT=1 cargo test --release -p clawhdf5-remote --test
|
||||
http -- --nocapture corpus`), requests as the test server counted
|
||||
them: the 621 corpus files that open read over HTTP exactly as through
|
||||
`File::open`. Open + list (every group's entries, every dataset's shape
|
||||
and type) of all of them: 640 requests, 55.5 MB of 254 MB; then reading
|
||||
each file's largest dataset under 64 MiB: 96 more (171 MB in all). The
|
||||
same work without a cache: 141 936 requests. Per file: A lists in 2
|
||||
requests (§2 predicted 2 blocks of 1 MiB), B in 1, C in 7 (its whole
|
||||
6.4 MB: 35 001 object headers spread over the file).
|
||||
|
||||
**M4 — wasm lazy loading (1–2 weeks).**
|
||||
- `clawhdf5-wasm`: `openUrl(url) -> Promise<H5File>` backed by `fetch` with a
|
||||
|
||||
Reference in New Issue
Block a user