docs: remote files after the adversarial review
CHANGELOG, the clawhdf5-remote and h5rs READMEs and the remote-files known issues: redirect rules, scaled timeouts (min_speed), URL redaction, claimed lengths never allocated (download, --max-download), a 200 for a small file accepted, and ObjectStoreStorage from any thread. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
+18
-1
@@ -31,6 +31,21 @@
|
||||
`408`/`429`/`5xx` and cut-off bodies are retried with exponential
|
||||
backoff; bodies are asked for with `Accept-Encoding: identity` and an
|
||||
encoded one is refused. The ranges of one call are fetched in parallel.
|
||||
A `200` answer to the first request whose body fits the range asked
|
||||
for is taken as the whole file (a server may answer so for a small
|
||||
file). Timeouts scale with the request: `HttpOptions::timeout` (30 s)
|
||||
to connect and to get the headers, plus the body's size at
|
||||
`HttpOptions::min_speed` (16 KiB/s), so a slow link is not cut off.
|
||||
Redirects: at most `HttpOptions::max_redirects` (5), never from `https`
|
||||
to `http`, and `HttpOptions::headers` are not sent to another origin.
|
||||
No error or `Debug` output contains a URL's userinfo or query values
|
||||
(`redact_url`; presigned URLs carry their signature there).
|
||||
- **Hostile lengths**: the length a server claims is never used to
|
||||
allocate. The cache's arithmetic is checked (a length near `u64::MAX`
|
||||
used to overflow), a read spanning more than the budget is fetched
|
||||
piece by piece with its output growing as data arrives, and
|
||||
`download(storage, max_bytes)` reads a whole file only up to a limit
|
||||
(`RemoteError::TooLarge` before any request otherwise).
|
||||
- **`ObjectStoreStorage`** reads one object of any `object_store` store,
|
||||
pinned by ETag (else version or modification time) and size. Each read
|
||||
runs on a small tokio runtime the storage owns while the caller waits,
|
||||
@@ -49,7 +64,9 @@
|
||||
requests; the tests hold it to at most 3.
|
||||
- **`h5rs` takes URLs** with the new `remote` feature (`remote-https` for
|
||||
`https://`): `ls`, `dump`, `stat` and `diff` read by range requests;
|
||||
`check` downloads the file whole, since it validates every byte. The
|
||||
`check` downloads the file whole, since it validates every byte, up to
|
||||
`--max-download N` (1 GiB by default). URLs are printed without their
|
||||
credentials. The
|
||||
tools now read through `File::storage` and the format crate's `*_in`
|
||||
functions; local output is unchanged.
|
||||
- **`File::storage()`** (facade) returns the file's bytes from the
|
||||
|
||||
Reference in New Issue
Block a user