osobhandClaude Opus 5.5
c5b2afbc35
It refused whenever Handle::try_current() was Ok, which is also the case inside spawn_blocking threads — so the workaround its own error message recommended failed the same way, and the backend could only be used from a bare std::thread in a tokio application. Reads are now spawned on the storage's own runtime and the caller waits on a channel: the future never runs on the caller's thread, so neither a spawn_blocking thread nor a current-thread runtime can deadlock or panic (a read inside a runtime blocks that thread, like any blocking call; the docs still recommend spawn_blocking there). Tests: a read in spawn_blocking of a multi-thread runtime and a read inside a current-thread runtime's task give File::open's values (both errors before). Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
100 lines
4.8 KiB
Markdown
100 lines
4.8 KiB
Markdown
# clawhdf5-remote
|
|
|
|
Read HDF5 files where they live — on an HTTP(S) server or in an object
|
|
store (S3, GCS, Azure) — with
|
|
[clawhdf5](../../README.md), without downloading them first.
|
|
|
|
```rust
|
|
let file = clawhdf5_remote::open_url("http://127.0.0.1:8000/data.h5")?;
|
|
let temps = file.dataset("/grid/temperature")?.read_f64()?;
|
|
```
|
|
|
|
The result is an ordinary `clawhdf5::File`: groups, datasets, attributes,
|
|
selections, variable-length data. Only the bytes an operation needs are
|
|
fetched, by `Range` requests, through a block cache.
|
|
|
|
## How it reads
|
|
|
|
- **Block cache** (`BlockCache`, mandatory for remote files; see
|
|
`docs/design/range-reads.md` §2): aligned blocks of 1 MiB by default, LRU
|
|
with a byte budget (64 MiB by default), the missing blocks of one read
|
|
fetched as runs of consecutive blocks (a gap of one block is fetched to
|
|
merge two runs), each request at most 8 MiB, the requests of one read in
|
|
parallel. A read that misses more than half the budget is not kept, so a
|
|
big dataset read does not evict the metadata. Readers on several threads
|
|
share one cache; a block is never fetched twice at once — a second reader
|
|
waits for the first one's request.
|
|
- **Opening costs one request**: a `GET` of the first block, whose
|
|
`Content-Range` gives the file's length. HDF5 files keep the superblock
|
|
and usually the root group's metadata there, so listing a small file
|
|
often needs nothing more.
|
|
- **Pinned to one version of the file**: a strong `ETag` is sent back as
|
|
`If-Match` (else `Last-Modified` as `If-Unmodified-Since`) and checked on
|
|
every response, as is the length. A file replaced while open is an error
|
|
(`RemoteError::FileChanged`), never a mix of old and new bytes. A server
|
|
with neither validator can only be checked by length;
|
|
`HttpOptions::require_validator` refuses it.
|
|
- **Servers that ignore `Range`** (answer `200` with the whole file) are
|
|
refused (`RemoteError::RangeNotSupported`) without reading the body,
|
|
unless `HttpOptions::allow_full_download` is set; then the file is
|
|
downloaded once and read from memory.
|
|
- **Retries**: connection failures, timeouts, `408`/`429`/`5xx` and bodies
|
|
that end early are retried with exponential backoff (3 retries, from
|
|
200 ms). Bodies are requested with `Accept-Encoding: identity`; an encoded
|
|
body is refused.
|
|
|
|
The zero-copy methods of `clawhdf5` (`read_raw_ref`, `read_*_zerocopy`,
|
|
`File::as_bytes`) borrow the whole file from memory, so they are errors
|
|
(`as_bytes` a panic; use `File::contiguous_bytes`) on a remote file.
|
|
|
|
## Features
|
|
|
|
| Feature | What | C code |
|
|
|---|---|---|
|
|
| `http` (default) | `http://` through `ureq`, no TLS | none |
|
|
| `https` | `https://` through rustls, ring provider, Mozilla roots | ring (C and assembly) |
|
|
| `object-store` | `ObjectStoreStorage` and `open_object` over any [`object_store`](https://docs.rs/object_store) store (in-memory, local files, or one you configure) | none |
|
|
| `s3`, `gcs`, `azure` | `s3://bucket/key`, `gs://bucket/key`, `az://container/key` in `open_url`, configured from the environment (`AWS_*`, `GOOGLE_*`, `AZURE_*`) as object_store's `from_env` builders read it | aws-lc-rs (object_store's cloud clients) |
|
|
|
|
The default build and `object-store` compile no C (`scripts/ci-test.sh`
|
|
checks both).
|
|
|
|
## Object stores
|
|
|
|
`object_store` is async; `Storage` is synchronous (parsing is CPU work).
|
|
`ObjectStoreStorage` owns a small tokio runtime (two worker threads): each
|
|
read runs there while the calling thread waits, so it works from any
|
|
thread, several at once — including `tokio::task::spawn_blocking` and code
|
|
inside another runtime (where `spawn_blocking` is still the better place,
|
|
since a read blocks the thread it is called on). The object is pinned by its ETag
|
|
(`If-Match`, and compared on every response), else its version or
|
|
modification time, and its size. The ranges of one read are fetched
|
|
concurrently (up to 8).
|
|
|
|
```rust
|
|
use std::sync::Arc;
|
|
use clawhdf5_remote::object_store::{memory::InMemory, ObjectStore};
|
|
let store: Arc<dyn ObjectStore> = Arc::new(InMemory::new());
|
|
// ... put a file at "data.h5" ...
|
|
let (file, cache) = clawhdf5_remote::open_object(store, "data.h5", &Default::default())?;
|
|
```
|
|
|
|
The tests use object_store's in-memory and local-file stores; no cloud
|
|
account is needed. The cloud schemes are only built (and unit-tested for
|
|
URL parsing) in CI, not run against a real bucket.
|
|
|
|
## Counting requests
|
|
|
|
```rust
|
|
use clawhdf5_remote::{storage_for_url, Options};
|
|
let storage = storage_for_url(url, &Options::default())?;
|
|
let file = clawhdf5::File::open_storage(storage.clone())?;
|
|
// ... read ...
|
|
let s = storage.stats(); // requests, bytes_fetched, hits, misses, cached_bytes, ...
|
|
```
|
|
|
|
`cargo run -p clawhdf5-remote --example range_server -- DIR` serves a
|
|
directory with range support (the server the tests use), and
|
|
`cargo run -p clawhdf5-remote --example read_url -- URL [DATASET]` lists a
|
|
file and prints what it cost.
|