clawhdf5/crates/clawhdf5-remote/README.md

# clawhdf5-remote

Read HDF5 files where they live — on an HTTP(S) server or in an object
store (S3, GCS, Azure) — with
[clawhdf5](../../README.md), without downloading them first.

```rust
let file = clawhdf5_remote::open_url("http://127.0.0.1:8000/data.h5")?;
let temps = file.dataset("/grid/temperature")?.read_f64()?;
```

The result is an ordinary `clawhdf5::File`: groups, datasets, attributes,
selections, variable-length data. Only the bytes an operation needs are
fetched, by `Range` requests, through a block cache.

## How it reads

- **Block cache** (`BlockCache`, mandatory for remote files; see
  `docs/design/range-reads.md` §2): aligned blocks of 1 MiB by default, LRU
  with a byte budget (64 MiB by default), the missing blocks of one read
  fetched as runs of consecutive blocks (a gap of one block is fetched to
  merge two runs), each request at most 8 MiB, the requests of one read in
  parallel. A read that misses more than half the budget is not kept, so a
  big dataset read does not evict the metadata. Readers on several threads
  share one cache; a block is never fetched twice at once — a second reader
  waits for the first one's request.
- **Opening costs one request**: a `GET` of the first block, whose
  `Content-Range` gives the file's length. HDF5 files keep the superblock
  and usually the root group's metadata there, so listing a small file
  often needs nothing more.
- **Pinned to one version of the file**: a strong `ETag` is sent back as
  `If-Match` (else `Last-Modified` as `If-Unmodified-Since`) and checked on
  every response, as is the length. A file replaced while open is an error
  (`RemoteError::FileChanged`), never a mix of old and new bytes. A server
  with neither validator can only be checked by length;
  `HttpOptions::require_validator` refuses it.
- **Servers that ignore `Range`** (answer `200` with the whole file) are
  refused (`RemoteError::RangeNotSupported`) without reading the body,
  unless `HttpOptions::allow_full_download` is set; then the file is
  downloaded once and read from memory.
  A `200` whose body is no longer than the range asked for is the whole
  (small) file, and is accepted.
- **Retries**: connection failures, timeouts, `408`/`429`/`5xx` and bodies
  that end early are retried with exponential backoff (3 retries, from
  200 ms). Bodies are requested with `Accept-Encoding: identity`; an encoded
  body is refused.
- **Timeouts** scale with the request: `HttpOptions::timeout` (30 s) to
  connect and to receive the headers, and for the body that plus its size
  at `HttpOptions::min_speed` (16 KiB/s) — a slow link is not cut off
  mid-block, a stalled connection still fails.
- **Redirects** are followed up to `HttpOptions::max_redirects` (5; 0
  refuses them), never from `https` to `http`. Once a redirect leaves the
  URL's origin (scheme, host, port), `HttpOptions::headers` (API keys,
  `Authorization`, cookies) are no longer sent.
- **Credentials stay out of messages**: every error and `Debug` output
  shows URLs through `redact_url` — no `user:password@`, query values
  replaced by `REDACTED` (a presigned S3/GCS URL's signature lives there).
- **Claimed lengths are not trusted**: nothing is allocated for the length
  a server reports; a read spanning more than the cache budget is fetched
  in pieces as data arrives, and `download(&storage, max_bytes)` reads a
  whole file only up to a limit (`DEFAULT_MAX_DOWNLOAD`, 1 GiB).

The zero-copy methods of `clawhdf5` (`read_raw_ref`, `read_*_zerocopy`,
`File::as_bytes`) borrow the whole file from memory, so they are errors
(`as_bytes` a panic; use `File::contiguous_bytes`) on a remote file.

## Features

| Feature | What | C code |
|---|---|---|
| `http` (default) | `http://` through `ureq`, no TLS | none |
| `https` | `https://` through rustls, ring provider, Mozilla roots | ring (C and assembly) |
| `object-store` | `ObjectStoreStorage` and `open_object` over any [`object_store`](https://docs.rs/object_store) store (in-memory, local files, or one you configure) | none |
| `s3`, `gcs`, `azure` | `s3://bucket/key`, `gs://bucket/key`, `az://container/key` in `open_url`, configured from the environment (`AWS_*`, `GOOGLE_*`, `AZURE_*`) as object_store's `from_env` builders read it | aws-lc-rs (object_store's cloud clients) |

The default build and `object-store` compile no C (`scripts/ci-test.sh`
checks both).

## Object stores

`object_store` is async; `Storage` is synchronous (parsing is CPU work).
`ObjectStoreStorage` owns a small tokio runtime (two worker threads): each
read runs there while the calling thread waits, so it works from any
thread, several at once — including `tokio::task::spawn_blocking` and code
inside another runtime (where `spawn_blocking` is still the better place,
since a read blocks the thread it is called on). The object is pinned by its ETag
(`If-Match`, and compared on every response), else its version or
modification time, and its size. The ranges of one read are fetched
concurrently (up to 8).

```rust
use std::sync::Arc;
use clawhdf5_remote::object_store::{memory::InMemory, ObjectStore};
let store: Arc<dyn ObjectStore> = Arc::new(InMemory::new());
// ... put a file at "data.h5" ...
let (file, cache) = clawhdf5_remote::open_object(store, "data.h5", &Default::default())?;
```

The tests use object_store's in-memory and local-file stores; no cloud
account is needed. The cloud schemes are only built (and unit-tested for
URL parsing) in CI, not run against a real bucket.

## Counting requests

```rust
use clawhdf5_remote::{storage_for_url, Options};
let storage = storage_for_url(url, &Options::default())?;
let file = clawhdf5::File::open_storage(storage.clone())?;
// ... read ...
let s = storage.stats(); // requests, bytes_fetched, hits, misses, cached_bytes, ...
```

`cargo run -p clawhdf5-remote --example range_server -- DIR` serves a
directory with range support (the server the tests use), and
`cargo run -p clawhdf5-remote --example read_url -- URL [DATASET]` lists a
file and prints what it cost.