Merge branch 'feat/p3-m3-remote' into feat/p3-remote-editor

# Conflicts:
#	crates/clawhdf5/tests/storage_equivalence.rs
This commit is contained in:
osobh
2026-09-26 19:17:32 -05:00
35 changed files with 5502 additions and 89 deletions
+42 -2
View File
@@ -5,8 +5,10 @@ change. Progress: M0 and M1 are done, and so is M2 (branch
`feat/p3-m2-raw-data`): every read path of the format crate works through
`Storage`, v2 B-trees, dense groups and raw data included, and
`File::open_storage` gives the facade's read API over any `Storage` (see
`CHANGELOG.md`, "Range reads, milestone M2"). M3 (a remote backend with
its block cache) is next. Every count in §1–§2 was
`CHANGELOG.md`, "Range reads, milestone M2"). M3 is done on branch
`feat/p3-m3-remote`: the `clawhdf5-remote` crate (block cache, HTTP(S),
object stores) and URLs in `h5rs` (see the M3 status below). M4 (wasm) is
next. Every count in §1–§2 was
taken on `tank` on 2026-09-26 at commit `de2a53f`, with the commands given
next to it. No timing numbers appear here on purpose: the machine was shared
with other build jobs when this was written.
@@ -462,6 +464,44 @@ fast path within benchmark noise.
§2; the page size for paged files; the first block prefetched on open) and
a request counter exposed for tests and users.
- Python bindings: `clawhdf5.File("s3://…")` / `https://` through it.
- *Status 2026-09-26:* done on branch `feat/p3-m3-remote`, except the
Python bindings, with these choices:
- A new crate, `clawhdf5-remote`, instead of a `remote` feature of
`clawhdf5-io`: `open_url` returns a `clawhdf5::File`, and `clawhdf5-io`
sits below the facade.
- HTTP(S) through `ureq` (`HttpStorage`), not object_store's HTTP store:
that one pulls reqwest with aws-lc-rs (C), while plain HTTP through
ureq builds no C, so it is the default feature; `https` adds rustls
with ring. S3/GCS/Azure go through `object_store` (`ObjectStoreStorage`,
features `s3`/`gcs`/`azure`, opt-in because of aws-lc-rs); the
`object-store` feature alone (in-memory, local files, a store you
build) is pure Rust. object_store is async: each read runs on a
two-thread tokio runtime of the storage's own while the caller waits,
so the caller's context (a plain thread, `spawn_blocking`, another
runtime) does not matter.
- `BlockCache` (any `Storage`): 1 MiB blocks and a 64 MiB LRU budget by
default, the first block fetched at open (for HTTP by the request
that learns the length), the missing blocks of one read fetched as
runs of consecutive blocks in one parallel batch, per-block in-flight
deduplication across threads, and reads that miss more than half the
budget not kept. The page size of paged files is not used as the block
size yet. `CacheStats` and `HttpStats` count requests and bytes.
- The file is pinned at open by ETag (else Last-Modified, or the object's
version) and length; a change is an error, not mixed data.
- `h5rs` (feature `remote`) reads through the new `File::storage()`, the
file's view as a `Storage`, so its parsing works on remote files; its
`check` downloads the file whole.
- Measured with `crates/clawhdf5-remote/tests/http.rs` (tank, 2026-09-26,
`CLAWHDF5_REMOTE_CORPUS=conformance/.cache/corpus
CLAWHDF5_REMOTE_REPORT=1 cargo test --release -p clawhdf5-remote --test
http -- --nocapture corpus`), requests as the test server counted
them: the 621 corpus files that open read over HTTP exactly as through
`File::open`. Open + list (every group's entries, every dataset's shape
and type) of all of them: 640 requests, 55.5 MB of 254 MB; then reading
each file's largest dataset under 64 MiB: 96 more (171 MB in all). The
same work without a cache: 141 936 requests. Per file: A lists in 2
requests (§2 predicted 2 blocks of 1 MiB), B in 1, C in 7 (its whole
6.4 MB: 35 001 object headers spread over the file).
**M4 — wasm lazy loading (1–2 weeks).**
- `clawhdf5-wasm`: `openUrl(url) -> Promise<H5File>` backed by `fetch` with a