clawhdf5-remote: object stores through object_store (S3, GCS, Azure)

ObjectStoreStorage (feature `object-store`, pure Rust) reads one object
of any object_store store by ranged get_opts, pinned at open by a head
request: If-Match with its ETag (and the ETag and size of every response
compared), else its version or modification time. A change is
RemoteError::FileChanged. object_store is async and Storage is not, so
the storage owns a small multi-threaded tokio runtime (two workers) and
blocks the calling thread on it; the ranges of one read_ranges call are
fetched concurrently (up to 8). From inside another tokio runtime it
refuses with RemoteError::Usage instead of blocking a worker, and it
shuts its runtime down in the background on drop so dropping it in async
code does not panic.

open_object(store, path, options) opens a file through a block cache
(first block prefetched); open_url accepts s3://, gs:// and az:// with
the `s3`, `gcs` and `azure` features, configured from the environment by
object_store's from_env builders. Those pull object_store's cloud clients
and aws-lc-rs (C), so they are opt-in; without them the URL is a clean
UnsupportedScheme error naming the feature.

Tests against object_store's in-memory and local-file stores (no cloud):
every fixture's transcript equals File::open's, a multi-block object is
fetched in coalesced block runs, an object replaced while open is an
error, and a missing object or a read from inside a runtime is a clean
error. ci-test.sh lints all backends, runs these tests (with s3 for its
URL parsing test) and checks object-store for C in the no-C step.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-26 17:15:06 -05:00
co-authored by Claude Opus 5.5
parent db2554dd81
commit 4ff3e40fea
8 changed files with 517 additions and 21 deletions
+5 -4
View File
@@ -474,10 +474,11 @@ fn bad_urls_and_statuses_are_clean_errors() {
open_url("no-scheme"),
Err(Error::Remote(RemoteError::InvalidUrl(_)))
));
assert!(matches!(
open_url("s3://bucket/key.h5"),
Err(Error::Remote(RemoteError::UnsupportedScheme(_)))
));
#[cfg(not(feature = "s3"))]
{
let e = open_url("s3://bucket/key.h5").unwrap_err().to_string();
assert!(e.contains("`s3` feature"), "{e}");
}
#[cfg(not(feature = "https"))]
{
let e = open_url("https://example.com/a.h5")