docs: remote files after the adversarial review
CHANGELOG, the clawhdf5-remote and h5rs READMEs and the remote-files known issues: redirect rules, scaled timeouts (min_speed), URL redaction, claimed lengths never allocated (download, --max-download), a 200 for a small file accepted, and ObjectStoreStorage from any thread. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
@@ -38,10 +38,27 @@ fetched, by `Range` requests, through a block cache.
|
||||
refused (`RemoteError::RangeNotSupported`) without reading the body,
|
||||
unless `HttpOptions::allow_full_download` is set; then the file is
|
||||
downloaded once and read from memory.
|
||||
A `200` whose body is no longer than the range asked for is the whole
|
||||
(small) file, and is accepted.
|
||||
- **Retries**: connection failures, timeouts, `408`/`429`/`5xx` and bodies
|
||||
that end early are retried with exponential backoff (3 retries, from
|
||||
200 ms). Bodies are requested with `Accept-Encoding: identity`; an encoded
|
||||
body is refused.
|
||||
- **Timeouts** scale with the request: `HttpOptions::timeout` (30 s) to
|
||||
connect and to receive the headers, and for the body that plus its size
|
||||
at `HttpOptions::min_speed` (16 KiB/s) — a slow link is not cut off
|
||||
mid-block, a stalled connection still fails.
|
||||
- **Redirects** are followed up to `HttpOptions::max_redirects` (5; 0
|
||||
refuses them), never from `https` to `http`. Once a redirect leaves the
|
||||
URL's origin (scheme, host, port), `HttpOptions::headers` (API keys,
|
||||
`Authorization`, cookies) are no longer sent.
|
||||
- **Credentials stay out of messages**: every error and `Debug` output
|
||||
shows URLs through `redact_url` — no `user:password@`, query values
|
||||
replaced by `REDACTED` (a presigned S3/GCS URL's signature lives there).
|
||||
- **Claimed lengths are not trusted**: nothing is allocated for the length
|
||||
a server reports; a read spanning more than the cache budget is fetched
|
||||
in pieces as data arrives, and `download(&storage, max_bytes)` reads a
|
||||
whole file only up to a limit (`DEFAULT_MAX_DOWNLOAD`, 1 GiB).
|
||||
|
||||
The zero-copy methods of `clawhdf5` (`read_raw_ref`, `read_*_zerocopy`,
|
||||
`File::as_bytes`) borrow the whole file from memory, so they are errors
|
||||
|
||||
@@ -42,7 +42,8 @@ h5rs diff local.h5 http://127.0.0.1:8000/file.h5
|
||||
A URL names the whole file (`FILE/OBJECT` suffixes are for local paths).
|
||||
`check` validates every byte, so it downloads a remote file whole first —
|
||||
up to `--max-download N` bytes (default 1 GiB), refusing a longer file
|
||||
before reading any of it.
|
||||
before reading any of it. URLs are printed without their credentials
|
||||
(userinfo, query string values).
|
||||
The output is the local file's (`tests/remote.rs` compares every
|
||||
subcommand).
|
||||
|
||||
|
||||
Reference in New Issue
Block a user