docs: remote files after the adversarial review

CHANGELOG, the clawhdf5-remote and h5rs READMEs and the remote-files
known issues: redirect rules, scaled timeouts (min_speed), URL redaction,
claimed lengths never allocated (download, --max-download), a 200 for a
small file accepted, and ObjectStoreStorage from any thread.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-26 18:37:05 -05:00
co-authored by Claude Opus 5.5
parent efb88f94e3
commit 0e98ffc498
4 changed files with 53 additions and 6 deletions
+17
View File
@@ -38,10 +38,27 @@ fetched, by `Range` requests, through a block cache.
refused (`RemoteError::RangeNotSupported`) without reading the body,
unless `HttpOptions::allow_full_download` is set; then the file is
downloaded once and read from memory.
A `200` whose body is no longer than the range asked for is the whole
(small) file, and is accepted.
- **Retries**: connection failures, timeouts, `408`/`429`/`5xx` and bodies
that end early are retried with exponential backoff (3 retries, from
200 ms). Bodies are requested with `Accept-Encoding: identity`; an encoded
body is refused.
- **Timeouts** scale with the request: `HttpOptions::timeout` (30 s) to
connect and to receive the headers, and for the body that plus its size
at `HttpOptions::min_speed` (16 KiB/s) — a slow link is not cut off
mid-block, a stalled connection still fails.
- **Redirects** are followed up to `HttpOptions::max_redirects` (5; 0
refuses them), never from `https` to `http`. Once a redirect leaves the
URL's origin (scheme, host, port), `HttpOptions::headers` (API keys,
`Authorization`, cookies) are no longer sent.
- **Credentials stay out of messages**: every error and `Debug` output
shows URLs through `redact_url` — no `user:password@`, query values
replaced by `REDACTED` (a presigned S3/GCS URL's signature lives there).
- **Claimed lengths are not trusted**: nothing is allocated for the length
a server reports; a read spanning more than the cache budget is fetched
in pieces as data arrives, and `download(&storage, max_bytes)` reads a
whole file only up to a limit (`DEFAULT_MAX_DOWNLOAD`, 1 GiB).
The zero-copy methods of `clawhdf5` (`read_raw_ref`, `read_*_zerocopy`,
`File::as_bytes`) borrow the whole file from memory, so they are errors
+2 -1
View File
@@ -42,7 +42,8 @@ h5rs diff local.h5 http://127.0.0.1:8000/file.h5
A URL names the whole file (`FILE/OBJECT` suffixes are for local paths).
`check` validates every byte, so it downloads a remote file whole first —
up to `--max-download N` bytes (default 1 GiB), refusing a longer file
before reading any of it.
before reading any of it. URLs are printed without their credentials
(userinfo, query string values).
The output is the local file's (`tests/remote.rs` compares every
subcommand).