Merge branch 'feat/p3-m3-remote' into feat/p3-remote-editor
# Conflicts: # crates/clawhdf5/tests/storage_equivalence.rs
This commit is contained in:
@@ -2,6 +2,80 @@
|
||||
|
||||
## Unreleased
|
||||
|
||||
### Range reads, milestone M3: remote files (2026-09-26)
|
||||
- **New crate `clawhdf5-remote`.** `open_url("http://host/file.h5")` gives
|
||||
a `clawhdf5::File` (through `File::open_storage`) that reads the file by
|
||||
HTTP `Range` requests; `storage_for_url` returns the cached storage to
|
||||
read its statistics. Features: `http` (default; `ureq` without TLS, no
|
||||
C), `https` (rustls with ring, which compiles C), `object-store` (any
|
||||
`object_store` store, pure Rust), `s3`/`gcs`/`azure` (`s3://`, `gs://`,
|
||||
`az://` URLs, configured from the environment; object_store's cloud
|
||||
clients use aws-lc-rs, C).
|
||||
- **`BlockCache`** wraps any `Storage`: aligned 1 MiB blocks (the size
|
||||
`docs/design/range-reads.md` §2 measured), LRU with a byte budget
|
||||
(64 MiB), the blocks one read misses fetched with one backend
|
||||
`read_ranges` call as runs of consecutive blocks (a one-block gap is
|
||||
fetched to merge two runs; at most 8 MiB per request). Readers on
|
||||
several threads share it without holding its lock across a request, and
|
||||
a block being fetched is waited for, not fetched again. A read that
|
||||
misses more than half the budget is not kept (a large dataset does not
|
||||
evict the metadata). A failed fetch is an error for every reader waiting
|
||||
on it and is not cached.
|
||||
- **`HttpStorage`**: opening is one ranged `GET` of the first block,
|
||||
whose `Content-Range` gives the length. The file is pinned by its strong
|
||||
`ETag` (`If-Match`) or else `Last-Modified` (`If-Unmodified-Since`), and
|
||||
its length, checked on every response: a file changed while open is
|
||||
`RemoteError::FileChanged`, never mixed data. A server that ignores
|
||||
`Range` is refused without reading the body unless
|
||||
`HttpOptions::allow_full_download`. Connection failures, timeouts,
|
||||
`408`/`429`/`5xx` and cut-off bodies are retried with exponential
|
||||
backoff; bodies are asked for with `Accept-Encoding: identity` and an
|
||||
encoded one is refused. The ranges of one call are fetched in parallel.
|
||||
A `200` answer to the first request whose body fits the range asked
|
||||
for is taken as the whole file (a server may answer so for a small
|
||||
file). Timeouts scale with the request: `HttpOptions::timeout` (30 s)
|
||||
to connect and to get the headers, plus the body's size at
|
||||
`HttpOptions::min_speed` (16 KiB/s), so a slow link is not cut off.
|
||||
Redirects: at most `HttpOptions::max_redirects` (5), never from `https`
|
||||
to `http`, and `HttpOptions::headers` are not sent to another origin.
|
||||
No error or `Debug` output contains a URL's userinfo or query values
|
||||
(`redact_url`; presigned URLs carry their signature there).
|
||||
- **Hostile lengths**: the length a server claims is never used to
|
||||
allocate. The cache's arithmetic is checked (a length near `u64::MAX`
|
||||
used to overflow), a read spanning more than the budget is fetched
|
||||
piece by piece with its output growing as data arrives, and
|
||||
`download(storage, max_bytes)` reads a whole file only up to a limit
|
||||
(`RemoteError::TooLarge` before any request otherwise).
|
||||
- **`ObjectStoreStorage`** reads one object of any `object_store` store,
|
||||
pinned by ETag (else version or modification time) and size. Each read
|
||||
runs on a small tokio runtime the storage owns while the caller waits,
|
||||
so it works from any thread, `spawn_blocking` and other runtimes
|
||||
included.
|
||||
`open_object(store, path, options)` opens a file through a block cache.
|
||||
- Counted on the conformance corpus (tank, 2026-09-26,
|
||||
`CLAWHDF5_REMOTE_CORPUS=conformance/.cache/corpus CLAWHDF5_REMOTE_REPORT=1
|
||||
cargo test --release -p clawhdf5-remote --test http -- --nocapture corpus`):
|
||||
the 621 files that open (254 MB) read over HTTP exactly as through
|
||||
`File::open`; opening and listing them all (every group's entries, every
|
||||
dataset's shape and type) took 640 requests and 55.5 MB, and then
|
||||
reading each file's largest dataset under 64 MiB 96 more requests
|
||||
(171 MB in all). Without the cache the same work is 141 936 requests.
|
||||
Listing the 7.7 MB IMERG file (file A of the design's §2) takes 2
|
||||
requests; the tests hold it to at most 3.
|
||||
- **`h5rs` takes URLs** with the new `remote` feature (`remote-https` for
|
||||
`https://`): `ls`, `dump`, `stat` and `diff` read by range requests;
|
||||
`check` downloads the file whole, since it validates every byte, up to
|
||||
`--max-download N` (1 GiB by default). URLs are printed without their
|
||||
credentials. The
|
||||
tools now read through `File::storage` and the format crate's `*_in`
|
||||
functions; local output is unchanged.
|
||||
- **`File::storage()`** (facade) returns the file's bytes from the
|
||||
superblock on (the `as_bytes` view, cache image laid over) as a
|
||||
`&(dyn Storage + Send + Sync)` for every backend, so code that parses a
|
||||
file itself works on remote files too.
|
||||
- **`VlResolver::element_in` / `string_element_in`** (format): the
|
||||
`element`/`string_element` lookups over any `Storage`.
|
||||
|
||||
### Range reads, milestone M2: raw data and `File::open_storage` (2026-09-26)
|
||||
- **`clawhdf5::File::open_storage(Arc<dyn Storage + Send + Sync>)`** opens
|
||||
a file served by any `clawhdf5_format::storage::Storage` and gives the
|
||||
|
||||
Reference in New Issue
Block a user