clawhdf5-remote: object stores through object_store (S3, GCS, Azure)
ObjectStoreStorage (feature `object-store`, pure Rust) reads one object of any object_store store by ranged get_opts, pinned at open by a head request: If-Match with its ETag (and the ETag and size of every response compared), else its version or modification time. A change is RemoteError::FileChanged. object_store is async and Storage is not, so the storage owns a small multi-threaded tokio runtime (two workers) and blocks the calling thread on it; the ranges of one read_ranges call are fetched concurrently (up to 8). From inside another tokio runtime it refuses with RemoteError::Usage instead of blocking a worker, and it shuts its runtime down in the background on drop so dropping it in async code does not panic. open_object(store, path, options) opens a file through a block cache (first block prefetched); open_url accepts s3://, gs:// and az:// with the `s3`, `gcs` and `azure` features, configured from the environment by object_store's from_env builders. Those pull object_store's cloud clients and aws-lc-rs (C), so they are opt-in; without them the URL is a clean UnsupportedScheme error naming the feature. Tests against object_store's in-memory and local-file stores (no cloud): every fixture's transcript equals File::open's, a multi-block object is fetched in coalesced block runs, an object replaced while open is an error, and a missing object or a read from inside a runtime is a clean error. ci-test.sh lints all backends, runs these tests (with s3 for its URL parsing test) and checks object-store for C in the no-C step. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
@@ -26,7 +26,7 @@ Cargo workspace with 19 crates under `crates/` (plus `libaec-sys`, an internal F
|
||||
| `clawhdf5-napi` | Node.js native addon bindings |
|
||||
| `clawhdf5-py` | PyO3 Python bindings |
|
||||
| `clawhdf5-wasm` | WebAssembly (wasm-bindgen) reader for the browser; demo in `examples/wasm-viewer/` |
|
||||
| `clawhdf5-remote` | Remote files: `open_url` over HTTP(S) range requests through a mandatory block cache (`BlockCache`) |
|
||||
| `clawhdf5-remote` | Remote files: `open_url` over HTTP(S) range requests and object stores (`object_store`: S3, GCS, Azure) through a mandatory block cache (`BlockCache`) |
|
||||
| `clawhdf5-bench` | Benchmark suite |
|
||||
|
||||
## Key Features
|
||||
@@ -165,8 +165,11 @@ Cargo workspace with 19 crates under `crates/` (plus `libaec-sys`, an internal F
|
||||
runs coalesced into parallel requests). `HttpStorage` pins the file by
|
||||
ETag/Last-Modified and length (a change is `RemoteError::FileChanged`),
|
||||
refuses servers that ignore `Range` unless a full download is allowed,
|
||||
and retries transient failures. Default build is plain HTTP with no C;
|
||||
`https` (rustls + ring) is opt-in. Tests run a std-only HTTP server
|
||||
and retries transient failures. `ObjectStoreStorage` (feature
|
||||
`object-store`, pure Rust) blocks on a small owned tokio runtime and
|
||||
refuses to run inside another runtime. Default build is plain HTTP with
|
||||
no C; `https` (rustls + ring) and `s3`/`gcs`/`azure` (aws-lc-rs) are
|
||||
opt-in. Tests run a std-only HTTP server
|
||||
(`tests/common/server.rs`, also the `range_server` example);
|
||||
`CLAWHDF5_REMOTE_CORPUS=conformance/.cache/corpus` compares every corpus
|
||||
file over HTTP with `File::open`.
|
||||
|
||||
Reference in New Issue
Block a user