docs: remote files after the adversarial review

CHANGELOG, the clawhdf5-remote and h5rs READMEs and the remote-files
known issues: redirect rules, scaled timeouts (min_speed), URL redaction,
claimed lengths never allocated (download, --max-download), a 200 for a
small file accepted, and ObjectStoreStorage from any thread.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

CHANGELOG.md

@@ -31,6 +31,21 @@
     `408`/`429`/`5xx` and cut-off bodies are retried with exponential
     backoff; bodies are asked for with `Accept-Encoding: identity` and an
     encoded one is refused. The ranges of one call are fetched in parallel.
+    A `200` answer to the first request whose body fits the range asked
+    for is taken as the whole file (a server may answer so for a small
+    file). Timeouts scale with the request: `HttpOptions::timeout` (30 s)
+    to connect and to get the headers, plus the body's size at
+    `HttpOptions::min_speed` (16 KiB/s), so a slow link is not cut off.
+    Redirects: at most `HttpOptions::max_redirects` (5), never from `https`
+    to `http`, and `HttpOptions::headers` are not sent to another origin.
+    No error or `Debug` output contains a URL's userinfo or query values
+    (`redact_url`; presigned URLs carry their signature there).
+  - **Hostile lengths**: the length a server claims is never used to
+    allocate. The cache's arithmetic is checked (a length near `u64::MAX`
+    used to overflow), a read spanning more than the budget is fetched
+    piece by piece with its output growing as data arrives, and
+    `download(storage, max_bytes)` reads a whole file only up to a limit
+    (`RemoteError::TooLarge` before any request otherwise).
   - **`ObjectStoreStorage`** reads one object of any `object_store` store,
     pinned by ETag (else version or modification time) and size. Each read
     runs on a small tokio runtime the storage owns while the caller waits,
@@ -49,7 +64,9 @@
     requests; the tests hold it to at most 3.
 - **`h5rs` takes URLs** with the new `remote` feature (`remote-https` for
   `https://`): `ls`, `dump`, `stat` and `diff` read by range requests;
-  `check` downloads the file whole, since it validates every byte. The
+  `check` downloads the file whole, since it validates every byte, up to
+  `--max-download N` (1 GiB by default). URLs are printed without their
+  credentials. The
   tools now read through `File::storage` and the format crate's `*_in`
   functions; local output is unchanged.
 - **`File::storage()`** (facade) returns the file's bytes from the

crates/clawhdf5-remote/README.md

@@ -38,10 +38,27 @@ fetched, by `Range` requests, through a block cache.
   refused (`RemoteError::RangeNotSupported`) without reading the body,
   unless `HttpOptions::allow_full_download` is set; then the file is
   downloaded once and read from memory.
+  A `200` whose body is no longer than the range asked for is the whole
+  (small) file, and is accepted.
 - **Retries**: connection failures, timeouts, `408`/`429`/`5xx` and bodies
   that end early are retried with exponential backoff (3 retries, from
   200 ms). Bodies are requested with `Accept-Encoding: identity`; an encoded
   body is refused.
+- **Timeouts** scale with the request: `HttpOptions::timeout` (30 s) to
+  connect and to receive the headers, and for the body that plus its size
+  at `HttpOptions::min_speed` (16 KiB/s) — a slow link is not cut off
+  mid-block, a stalled connection still fails.
+- **Redirects** are followed up to `HttpOptions::max_redirects` (5; 0
+  refuses them), never from `https` to `http`. Once a redirect leaves the
+  URL's origin (scheme, host, port), `HttpOptions::headers` (API keys,
+  `Authorization`, cookies) are no longer sent.
+- **Credentials stay out of messages**: every error and `Debug` output
+  shows URLs through `redact_url` — no `user:password@`, query values
+  replaced by `REDACTED` (a presigned S3/GCS URL's signature lives there).
+- **Claimed lengths are not trusted**: nothing is allocated for the length
+  a server reports; a read spanning more than the cache budget is fetched
+  in pieces as data arrives, and `download(&storage, max_bytes)` reads a
+  whole file only up to a limit (`DEFAULT_MAX_DOWNLOAD`, 1 GiB).
 
 The zero-copy methods of `clawhdf5` (`read_raw_ref`, `read_*_zerocopy`,
 `File::as_bytes`) borrow the whole file from memory, so they are errors

crates/clawhdf5-tools/README.md

@@ -42,7 +42,8 @@ h5rs diff local.h5 http://127.0.0.1:8000/file.h5
 A URL names the whole file (`FILE/OBJECT` suffixes are for local paths).
 `check` validates every byte, so it downloads a remote file whole first —
 up to `--max-download N` bytes (default 1 GiB), refusing a longer file
-before reading any of it.
+before reading any of it. URLs are printed without their credentials
+(userinfo, query string values).
 The output is the local file's (`tests/remote.rs` compares every
 subcommand).
 

docs/known-issues.md

@@ -829,11 +829,23 @@ cache, but:
   ETag (`W/"…"`) cannot be sent as `If-Match`, so it counts as none.
 - **Credentials:** HTTP takes extra headers (`HttpOptions::headers`, e.g.
   `Authorization`); `h5rs` has no option for them. The cloud stores read
-  credentials from the environment only.
+  credentials from the environment only. Messages and `Debug` output show
+  URLs through `redact_url` (no userinfo, query values `REDACTED`);
+  `HttpStorage::url()` returns the URL as given and must not be logged.
+- **Redirects:** at most `HttpOptions::max_redirects` (5) per request,
+  never from `https` to `http`, and the custom headers are not sent once a
+  redirect leaves the URL's origin. The redirect target is not remembered:
+  every request of a redirected file costs its hops again.
+- **Timeouts:** ureq has no idle timeout, only a total one for the body,
+  so the body's budget is `timeout + size / min_speed` (30 s + 16 KiB/s
+  by default). A connection that stalls mid-body is detected only when
+  that budget runs out (94 s for a 1 MiB block, 9 min for an 8 MiB run).
 - Each `ObjectStoreStorage` owns a tokio runtime with two worker threads.
-- `h5rs check` downloads a remote file whole (it validates every byte), and
+  A read called from inside another runtime blocks that runtime's thread
-  a URL cannot carry a `FILE/OBJECT` suffix; `h5rs` uses the default cache
+  for its duration (it works, but `spawn_blocking` is the better place).
-  settings.
+- `h5rs check` downloads a remote file whole (it validates every byte), up
+  to `--max-download` (1 GiB by default), and a URL cannot carry a
+  `FILE/OBJECT` suffix; `h5rs` uses the default cache settings.
 - The zero-copy methods and `File::as_bytes` are unavailable on a remote
   file (see the range-read limits above).