Merge branch 'feat/p3-m3-remote' into feat/p3-remote-editor

# Conflicts:
#	crates/clawhdf5/tests/storage_equivalence.rs
This commit is contained in:
osobh
2026-09-26 19:17:32 -05:00
35 changed files with 5502 additions and 89 deletions
+74
View File
@@ -2,6 +2,80 @@
## Unreleased
### Range reads, milestone M3: remote files (2026-09-26)
- **New crate `clawhdf5-remote`.** `open_url("http://host/file.h5")` gives
a `clawhdf5::File` (through `File::open_storage`) that reads the file by
HTTP `Range` requests; `storage_for_url` returns the cached storage to
read its statistics. Features: `http` (default; `ureq` without TLS, no
C), `https` (rustls with ring, which compiles C), `object-store` (any
`object_store` store, pure Rust), `s3`/`gcs`/`azure` (`s3://`, `gs://`,
`az://` URLs, configured from the environment; object_store's cloud
clients use aws-lc-rs, C).
- **`BlockCache`** wraps any `Storage`: aligned 1 MiB blocks (the size
`docs/design/range-reads.md` §2 measured), LRU with a byte budget
(64 MiB), the blocks one read misses fetched with one backend
`read_ranges` call as runs of consecutive blocks (a one-block gap is
fetched to merge two runs; at most 8 MiB per request). Readers on
several threads share it without holding its lock across a request, and
a block being fetched is waited for, not fetched again. A read that
misses more than half the budget is not kept (a large dataset does not
evict the metadata). A failed fetch is an error for every reader waiting
on it and is not cached.
- **`HttpStorage`**: opening is one ranged `GET` of the first block,
whose `Content-Range` gives the length. The file is pinned by its strong
`ETag` (`If-Match`) or else `Last-Modified` (`If-Unmodified-Since`), and
its length, checked on every response: a file changed while open is
`RemoteError::FileChanged`, never mixed data. A server that ignores
`Range` is refused without reading the body unless
`HttpOptions::allow_full_download`. Connection failures, timeouts,
`408`/`429`/`5xx` and cut-off bodies are retried with exponential
backoff; bodies are asked for with `Accept-Encoding: identity` and an
encoded one is refused. The ranges of one call are fetched in parallel.
A `200` answer to the first request whose body fits the range asked
for is taken as the whole file (a server may answer so for a small
file). Timeouts scale with the request: `HttpOptions::timeout` (30 s)
to connect and to get the headers, plus the body's size at
`HttpOptions::min_speed` (16 KiB/s), so a slow link is not cut off.
Redirects: at most `HttpOptions::max_redirects` (5), never from `https`
to `http`, and `HttpOptions::headers` are not sent to another origin.
No error or `Debug` output contains a URL's userinfo or query values
(`redact_url`; presigned URLs carry their signature there).
- **Hostile lengths**: the length a server claims is never used to
allocate. The cache's arithmetic is checked (a length near `u64::MAX`
used to overflow), a read spanning more than the budget is fetched
piece by piece with its output growing as data arrives, and
`download(storage, max_bytes)` reads a whole file only up to a limit
(`RemoteError::TooLarge` before any request otherwise).
- **`ObjectStoreStorage`** reads one object of any `object_store` store,
pinned by ETag (else version or modification time) and size. Each read
runs on a small tokio runtime the storage owns while the caller waits,
so it works from any thread, `spawn_blocking` and other runtimes
included.
`open_object(store, path, options)` opens a file through a block cache.
- Counted on the conformance corpus (tank, 2026-09-26,
`CLAWHDF5_REMOTE_CORPUS=conformance/.cache/corpus CLAWHDF5_REMOTE_REPORT=1
cargo test --release -p clawhdf5-remote --test http -- --nocapture corpus`):
the 621 files that open (254 MB) read over HTTP exactly as through
`File::open`; opening and listing them all (every group's entries, every
dataset's shape and type) took 640 requests and 55.5 MB, and then
reading each file's largest dataset under 64 MiB 96 more requests
(171 MB in all). Without the cache the same work is 141 936 requests.
Listing the 7.7 MB IMERG file (file A of the design's §2) takes 2
requests; the tests hold it to at most 3.
- **`h5rs` takes URLs** with the new `remote` feature (`remote-https` for
`https://`): `ls`, `dump`, `stat` and `diff` read by range requests;
`check` downloads the file whole, since it validates every byte, up to
`--max-download N` (1 GiB by default). URLs are printed without their
credentials. The
tools now read through `File::storage` and the format crate's `*_in`
functions; local output is unchanged.
- **`File::storage()`** (facade) returns the file's bytes from the
superblock on (the `as_bytes` view, cache image laid over) as a
`&(dyn Storage + Send + Sync)` for every backend, so code that parses a
file itself works on remote files too.
- **`VlResolver::element_in` / `string_element_in`** (format): the
`element`/`string_element` lookups over any `Storage`.
### Range reads, milestone M2: raw data and `File::open_storage` (2026-09-26)
- **`clawhdf5::File::open_storage(Arc<dyn Storage + Send + Sync>)`** opens
a file served by any `clawhdf5_format::storage::Storage` and gives the