docs: remote files after the adversarial review

CHANGELOG, the clawhdf5-remote and h5rs READMEs and the remote-files
known issues: redirect rules, scaled timeouts (min_speed), URL redaction,
claimed lengths never allocated (download, --max-download), a 200 for a
small file accepted, and ObjectStoreStorage from any thread.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-26 18:37:05 -05:00
co-authored by Claude Opus 5.5
parent efb88f94e3
commit 0e98ffc498
4 changed files with 53 additions and 6 deletions
+18 -1
View File
@@ -31,6 +31,21 @@
`408`/`429`/`5xx` and cut-off bodies are retried with exponential
backoff; bodies are asked for with `Accept-Encoding: identity` and an
encoded one is refused. The ranges of one call are fetched in parallel.
A `200` answer to the first request whose body fits the range asked
for is taken as the whole file (a server may answer so for a small
file). Timeouts scale with the request: `HttpOptions::timeout` (30 s)
to connect and to get the headers, plus the body's size at
`HttpOptions::min_speed` (16 KiB/s), so a slow link is not cut off.
Redirects: at most `HttpOptions::max_redirects` (5), never from `https`
to `http`, and `HttpOptions::headers` are not sent to another origin.
No error or `Debug` output contains a URL's userinfo or query values
(`redact_url`; presigned URLs carry their signature there).
- **Hostile lengths**: the length a server claims is never used to
allocate. The cache's arithmetic is checked (a length near `u64::MAX`
used to overflow), a read spanning more than the budget is fetched
piece by piece with its output growing as data arrives, and
`download(storage, max_bytes)` reads a whole file only up to a limit
(`RemoteError::TooLarge` before any request otherwise).
- **`ObjectStoreStorage`** reads one object of any `object_store` store,
pinned by ETag (else version or modification time) and size. Each read
runs on a small tokio runtime the storage owns while the caller waits,
@@ -49,7 +64,9 @@
requests; the tests hold it to at most 3.
- **`h5rs` takes URLs** with the new `remote` feature (`remote-https` for
`https://`): `ls`, `dump`, `stat` and `diff` read by range requests;
`check` downloads the file whole, since it validates every byte. The
`check` downloads the file whole, since it validates every byte, up to
`--max-download N` (1 GiB by default). URLs are printed without their
credentials. The
tools now read through `File::storage` and the format crate's `*_in`
functions; local output is unchanged.
- **`File::storage()`** (facade) returns the file's bytes from the