clawhdf5-remote: object stores through object_store (S3, GCS, Azure)
ObjectStoreStorage (feature `object-store`, pure Rust) reads one object of any object_store store by ranged get_opts, pinned at open by a head request: If-Match with its ETag (and the ETag and size of every response compared), else its version or modification time. A change is RemoteError::FileChanged. object_store is async and Storage is not, so the storage owns a small multi-threaded tokio runtime (two workers) and blocks the calling thread on it; the ranges of one read_ranges call are fetched concurrently (up to 8). From inside another tokio runtime it refuses with RemoteError::Usage instead of blocking a worker, and it shuts its runtime down in the background on drop so dropping it in async code does not panic. open_object(store, path, options) opens a file through a block cache (first block prefetched); open_url accepts s3://, gs:// and az:// with the `s3`, `gcs` and `azure` features, configured from the environment by object_store's from_env builders. Those pull object_store's cloud clients and aws-lc-rs (C), so they are opt-in; without them the URL is a clean UnsupportedScheme error naming the feature. Tests against object_store's in-memory and local-file stores (no cloud): every fixture's transcript equals File::open's, a multi-block object is fetched in coalesced block runs, an object replaced while open is an error, and a missing object or a read from inside a runtime is a clean error. ci-test.sh lints all backends, runs these tests (with s3 for its URL parsing test) and checks object-store for C in the no-C step. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
@@ -474,10 +474,11 @@ fn bad_urls_and_statuses_are_clean_errors() {
|
||||
open_url("no-scheme"),
|
||||
Err(Error::Remote(RemoteError::InvalidUrl(_)))
|
||||
));
|
||||
assert!(matches!(
|
||||
open_url("s3://bucket/key.h5"),
|
||||
Err(Error::Remote(RemoteError::UnsupportedScheme(_)))
|
||||
));
|
||||
#[cfg(not(feature = "s3"))]
|
||||
{
|
||||
let e = open_url("s3://bucket/key.h5").unwrap_err().to_string();
|
||||
assert!(e.contains("`s3` feature"), "{e}");
|
||||
}
|
||||
#[cfg(not(feature = "https"))]
|
||||
{
|
||||
let e = open_url("https://example.com/a.h5")
|
||||
|
||||
Reference in New Issue
Block a user