diff --git a/CHANGELOG.md b/CHANGELOG.md index 344afbf..a281969 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -31,6 +31,21 @@ `408`/`429`/`5xx` and cut-off bodies are retried with exponential backoff; bodies are asked for with `Accept-Encoding: identity` and an encoded one is refused. The ranges of one call are fetched in parallel. + A `200` answer to the first request whose body fits the range asked + for is taken as the whole file (a server may answer so for a small + file). Timeouts scale with the request: `HttpOptions::timeout` (30 s) + to connect and to get the headers, plus the body's size at + `HttpOptions::min_speed` (16 KiB/s), so a slow link is not cut off. + Redirects: at most `HttpOptions::max_redirects` (5), never from `https` + to `http`, and `HttpOptions::headers` are not sent to another origin. + No error or `Debug` output contains a URL's userinfo or query values + (`redact_url`; presigned URLs carry their signature there). + - **Hostile lengths**: the length a server claims is never used to + allocate. The cache's arithmetic is checked (a length near `u64::MAX` + used to overflow), a read spanning more than the budget is fetched + piece by piece with its output growing as data arrives, and + `download(storage, max_bytes)` reads a whole file only up to a limit + (`RemoteError::TooLarge` before any request otherwise). - **`ObjectStoreStorage`** reads one object of any `object_store` store, pinned by ETag (else version or modification time) and size. Each read runs on a small tokio runtime the storage owns while the caller waits, @@ -49,7 +64,9 @@ requests; the tests hold it to at most 3. - **`h5rs` takes URLs** with the new `remote` feature (`remote-https` for `https://`): `ls`, `dump`, `stat` and `diff` read by range requests; - `check` downloads the file whole, since it validates every byte. The + `check` downloads the file whole, since it validates every byte, up to + `--max-download N` (1 GiB by default). URLs are printed without their + credentials. The tools now read through `File::storage` and the format crate's `*_in` functions; local output is unchanged. - **`File::storage()`** (facade) returns the file's bytes from the diff --git a/crates/clawhdf5-remote/README.md b/crates/clawhdf5-remote/README.md index c64c7c6..cd663b4 100644 --- a/crates/clawhdf5-remote/README.md +++ b/crates/clawhdf5-remote/README.md @@ -38,10 +38,27 @@ fetched, by `Range` requests, through a block cache. refused (`RemoteError::RangeNotSupported`) without reading the body, unless `HttpOptions::allow_full_download` is set; then the file is downloaded once and read from memory. + A `200` whose body is no longer than the range asked for is the whole + (small) file, and is accepted. - **Retries**: connection failures, timeouts, `408`/`429`/`5xx` and bodies that end early are retried with exponential backoff (3 retries, from 200 ms). Bodies are requested with `Accept-Encoding: identity`; an encoded body is refused. +- **Timeouts** scale with the request: `HttpOptions::timeout` (30 s) to + connect and to receive the headers, and for the body that plus its size + at `HttpOptions::min_speed` (16 KiB/s) — a slow link is not cut off + mid-block, a stalled connection still fails. +- **Redirects** are followed up to `HttpOptions::max_redirects` (5; 0 + refuses them), never from `https` to `http`. Once a redirect leaves the + URL's origin (scheme, host, port), `HttpOptions::headers` (API keys, + `Authorization`, cookies) are no longer sent. +- **Credentials stay out of messages**: every error and `Debug` output + shows URLs through `redact_url` — no `user:password@`, query values + replaced by `REDACTED` (a presigned S3/GCS URL's signature lives there). +- **Claimed lengths are not trusted**: nothing is allocated for the length + a server reports; a read spanning more than the cache budget is fetched + in pieces as data arrives, and `download(&storage, max_bytes)` reads a + whole file only up to a limit (`DEFAULT_MAX_DOWNLOAD`, 1 GiB). The zero-copy methods of `clawhdf5` (`read_raw_ref`, `read_*_zerocopy`, `File::as_bytes`) borrow the whole file from memory, so they are errors diff --git a/crates/clawhdf5-tools/README.md b/crates/clawhdf5-tools/README.md index fc120ad..24ecb9c 100644 --- a/crates/clawhdf5-tools/README.md +++ b/crates/clawhdf5-tools/README.md @@ -42,7 +42,8 @@ h5rs diff local.h5 http://127.0.0.1:8000/file.h5 A URL names the whole file (`FILE/OBJECT` suffixes are for local paths). `check` validates every byte, so it downloads a remote file whole first — up to `--max-download N` bytes (default 1 GiB), refusing a longer file -before reading any of it. +before reading any of it. URLs are printed without their credentials +(userinfo, query string values). The output is the local file's (`tests/remote.rs` compares every subcommand). diff --git a/docs/known-issues.md b/docs/known-issues.md index 860e1eb..7275020 100644 --- a/docs/known-issues.md +++ b/docs/known-issues.md @@ -829,11 +829,23 @@ cache, but: ETag (`W/"…"`) cannot be sent as `If-Match`, so it counts as none. - **Credentials:** HTTP takes extra headers (`HttpOptions::headers`, e.g. `Authorization`); `h5rs` has no option for them. The cloud stores read - credentials from the environment only. + credentials from the environment only. Messages and `Debug` output show + URLs through `redact_url` (no userinfo, query values `REDACTED`); + `HttpStorage::url()` returns the URL as given and must not be logged. +- **Redirects:** at most `HttpOptions::max_redirects` (5) per request, + never from `https` to `http`, and the custom headers are not sent once a + redirect leaves the URL's origin. The redirect target is not remembered: + every request of a redirected file costs its hops again. +- **Timeouts:** ureq has no idle timeout, only a total one for the body, + so the body's budget is `timeout + size / min_speed` (30 s + 16 KiB/s + by default). A connection that stalls mid-body is detected only when + that budget runs out (94 s for a 1 MiB block, 9 min for an 8 MiB run). - Each `ObjectStoreStorage` owns a tokio runtime with two worker threads. -- `h5rs check` downloads a remote file whole (it validates every byte), and - a URL cannot carry a `FILE/OBJECT` suffix; `h5rs` uses the default cache - settings. + A read called from inside another runtime blocks that runtime's thread + for its duration (it works, but `spawn_blocking` is the better place). +- `h5rs check` downloads a remote file whole (it validates every byte), up + to `--max-download` (1 GiB by default), and a URL cannot carry a + `FILE/OBJECT` suffix; `h5rs` uses the default cache settings. - The zero-copy methods and `File::as_bytes` are unavailable on a remote file (see the range-read limits above).