# clawhdf5-remote

Read HDF5 files where they live — on an HTTP(S) server — with
[clawhdf5](../../README.md), without downloading them first.

```rust
let file = clawhdf5_remote::open_url("http://127.0.0.1:8000/data.h5")?;
let temps = file.dataset("/grid/temperature")?.read_f64()?;
```

The result is an ordinary `clawhdf5::File`: groups, datasets, attributes,
selections, variable-length data. Only the bytes an operation needs are
fetched, by `Range` requests, through a block cache.

## How it reads

- **Block cache** (`BlockCache`, mandatory for remote files; see
  `docs/design/range-reads.md` §2): aligned blocks of 1 MiB by default, LRU
  with a byte budget (64 MiB by default), the missing blocks of one read
  fetched as runs of consecutive blocks (a gap of one block is fetched to
  merge two runs), each request at most 8 MiB, the requests of one read in
  parallel. A read that misses more than half the budget is not kept, so a
  big dataset read does not evict the metadata. Readers on several threads
  share one cache; a block is never fetched twice at once — a second reader
  waits for the first one's request.
- **Opening costs one request**: a `GET` of the first block, whose
  `Content-Range` gives the file's length. HDF5 files keep the superblock
  and usually the root group's metadata there, so listing a small file
  often needs nothing more.
- **Pinned to one version of the file**: a strong `ETag` is sent back as
  `If-Match` (else `Last-Modified` as `If-Unmodified-Since`) and checked on
  every response, as is the length. A file replaced while open is an error
  (`RemoteError::FileChanged`), never a mix of old and new bytes. A server
  with neither validator can only be checked by length;
  `HttpOptions::require_validator` refuses it.
- **Servers that ignore `Range`** (answer `200` with the whole file) are
  refused (`RemoteError::RangeNotSupported`) without reading the body,
  unless `HttpOptions::allow_full_download` is set; then the file is
  downloaded once and read from memory.
- **Retries**: connection failures, timeouts, `408`/`429`/`5xx` and bodies
  that end early are retried with exponential backoff (3 retries, from
  200 ms). Bodies are requested with `Accept-Encoding: identity`; an encoded
  body is refused.

The zero-copy methods of `clawhdf5` (`read_raw_ref`, `read_*_zerocopy`,
`File::as_bytes`) borrow the whole file from memory, so they are errors
(`as_bytes` a panic; use `File::contiguous_bytes`) on a remote file.

## Features

| Feature | What | C code |
|---|---|---|
| `http` (default) | `http://` through `ureq`, no TLS | none |
| `https` | `https://` through rustls, ring provider, Mozilla roots | ring (C and assembly) |

The default build compiles no C (`scripts/ci-test.sh` checks it).

## Counting requests

```rust
use clawhdf5_remote::{storage_for_url, Options};
let storage = storage_for_url(url, &Options::default())?;
let file = clawhdf5::File::open_storage(storage.clone())?;
// ... read ...
let s = storage.stats(); // requests, bytes_fetched, hits, misses, cached_bytes, ...
```

`cargo run -p clawhdf5-remote --example range_server -- DIR` serves a
directory with range support (the server the tests use), and
`cargo run -p clawhdf5-remote --example read_url -- URL [DATASET]` lists a
file and prints what it cost.