clawhdf5-remote: block cache and HTTP range reads (open_url)

Range-read milestone M3, first half: a new crate with the block cache the
design makes mandatory for remote files and an HTTP backend, so
open_url("http://...") gives a clawhdf5::File over File::open_storage.

BlockCache wraps any Storage: aligned blocks (1 MiB by default, the size
docs/design/range-reads.md section 2 measured), LRU with a byte budget,
the missing blocks of one read_at/read_ranges fetched with one backend
read_ranges call as runs of consecutive blocks (a one-block gap filled to
merge runs, each request at most 8 MiB), and reads that miss more than
half the budget not kept. Thread-safe without holding the lock across a
fetch: a block being fetched is in flight, a second reader waits for it
instead of fetching it again, and a failed fetch fails its waiters and is
not cached. A backend holding the file in memory passes through.

HttpStorage (ureq, no TLS by default; `https` adds rustls with ring):
opening is one ranged GET of the first block, whose Content-Range gives
the length (the cache keeps the bytes). The file is pinned by a strong
ETag (If-Match), else Last-Modified (If-Unmodified-Since), and its length,
checked on every response: a change is RemoteError::FileChanged, never
mixed data. A server that ignores Range is refused without reading the
body unless a full download is allowed. Connection errors, timeouts,
408/429/5xx and short bodies are retried with exponential backoff;
Accept-Encoding: identity, and an encoded body is refused. read_ranges
fetches its ranges in parallel.

Tests (a std-only HTTP/1.1 server in tests/common/server.rs, also the
range_server example): every fixture read over HTTP gives File::open's
transcript (CLAWHDF5_REMOTE_CORPUS adds the conformance corpus), with
request counts per file with and without the cache; an h5py-written file
against libhdf5's values; a multi-block file fetched in whole blocks, each
once; a server ignoring Range; a file replaced mid-read (ETag,
Last-Modified, length only); truncated bodies and 503s (retried, then an
error, never cached); a slow server with 8 concurrent readers (no block
fetched twice); bad URLs, 404, encoded bodies, non-HDF5 data. The cache
has unit tests for coalescing, splitting, LRU order, large reads,
failures and concurrent in-flight dedup.

ci-test.sh: clawhdf5-remote joins the no-C default-build check, and its
https feature is linted.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

crates/clawhdf5-remote/src/lib.rs

@@ -0,0 +1,127 @@
+//! Read HDF5 files where they are — on an HTTP(S) server or in an object
+//! store — without downloading them first.
+//!
+//! This is milestone M3 of `docs/design/range-reads.md`: remote backends for
+//! [`clawhdf5::File::open_storage`], each read through a mandatory
+//! [`BlockCache`].
+//!
+//! ```no_run
+//! let file = clawhdf5_remote::open_url("http://127.0.0.1:8000/data.h5")?;
+//! let temperature = file.dataset("/grid/temperature")?.read_f64()?;
+//! # Ok::<(), Box<dyn std::error::Error>>(())
+//! ```
+//!
+//! - [`open_url`] / [`open_url_with`]: `http://` (default feature `http`),
+//!   `https://` (feature `https`) → a [`clawhdf5::File`] with the whole
+//!   read API.
+//! - [`storage_for_url`] gives the cached storage itself, to open with
+//!   [`clawhdf5::File::open_storage`] and to read its [`CacheStats`].
+//! - [`HttpStorage`] (range `GET`s, pinned by ETag/Last-Modified, retried
+//!   with backoff), and [`BlockCache`] over any
+//!   [`Storage`](clawhdf5_format::storage::Storage).
+//!
+//! What costs what: opening costs one request (it also fetches the first
+//! block, 1 MiB by default); listing a file whose metadata sits in its
+//! first blocks costs nothing more; reading a chunked dataset costs one
+//! parallel batch of requests for the blocks holding its chunk index, then
+//! one for its chunks. The zero-copy methods of `clawhdf5`
+//! (`read_raw_ref`, `read_*_zerocopy`, `File::as_bytes`) need the file in
+//! memory and are errors (`as_bytes` a panic) on a remote file.
+
+#![warn(missing_docs)]
+
+pub mod cache;
+pub mod error;
+#[cfg(feature = "http")]
+pub mod http;
+
+use std::sync::Arc;
+
+use clawhdf5::File;
+use clawhdf5_format::storage::Storage;
+
+pub use cache::{BlockCache, CacheConfig, CacheStats};
+pub use error::{Error, RemoteError};
+#[cfg(feature = "http")]
+pub use http::{HttpOptions, HttpStats, HttpStorage};
+
+/// A backend a [`BlockCache`] can read through.
+pub type Backend = Box<dyn Storage + Send + Sync>;
+
+/// The storage [`storage_for_url`] returns: a block cache over the URL's
+/// backend.
+pub type RemoteStorage = BlockCache<Backend>;
+
+/// Settings of [`open_url_with`] and [`storage_for_url`].
+#[derive(Debug, Clone, Default)]
+pub struct Options {
+    /// The block cache.
+    pub cache: CacheConfig,
+    /// HTTP(S) requests.
+    #[cfg(feature = "http")]
+    pub http: HttpOptions,
+}
+
+/// Open the HDF5 file at `url` with default [`Options`].
+///
+/// `http://…` needs the (default) `http` feature, `https://…` the `https`
+/// feature.
+pub fn open_url(url: &str) -> Result<File, Error> {
+    open_url_with(url, &Options::default())
+}
+
+/// [`open_url`] with explicit [`Options`].
+pub fn open_url_with(url: &str, options: &Options) -> Result<File, Error> {
+    let storage = storage_for_url(url, options)?;
+    Ok(File::open_storage(storage)?)
+}
+
+/// The cached storage for `url`, with the first block already fetched:
+/// open it with [`clawhdf5::File::open_storage`] (a clone of the `Arc`),
+/// and read its [`BlockCache::stats`] as you go.
+pub fn storage_for_url(url: &str, options: &Options) -> Result<Arc<RemoteStorage>, Error> {
+    let scheme = url
+        .split_once("://")
+        .map(|(s, _)| s.to_ascii_lowercase())
+        .ok_or_else(|| RemoteError::InvalidUrl(format!("{url}: no scheme")))?;
+    match scheme.as_str() {
+        "http" | "https" => http_storage(url, options),
+        "s3" | "s3a" | "gs" | "az" | "azure" | "abfs" | "abfss" | "adl" => {
+            cloud_storage(url, &scheme, options)
+        }
+        _ => Err(RemoteError::UnsupportedScheme(url.to_string()).into()),
+    }
+}
+
+#[cfg(feature = "http")]
+fn http_storage(url: &str, options: &Options) -> Result<Arc<RemoteStorage>, Error> {
+    let mut http = options.http.clone();
+    http.first_request = http.first_request.max(options.cache.block_size.max(1));
+    let (storage, first) = HttpStorage::open(url, http)?;
+    let cache = BlockCache::new(Box::new(storage) as Backend, options.cache.clone());
+    cache.insert(0, &first);
+    Ok(Arc::new(cache))
+}
+
+#[cfg(not(feature = "http"))]
+fn http_storage(url: &str, _options: &Options) -> Result<Arc<RemoteStorage>, Error> {
+    Err(RemoteError::UnsupportedScheme(format!(
+        "{url}: http(s):// needs the `http` feature of clawhdf5-remote"
+    ))
+    .into())
+}
+
+fn cloud_storage(url: &str, scheme: &str, _options: &Options) -> Result<Arc<RemoteStorage>, Error> {
+    Err(RemoteError::UnsupportedScheme(format!("{url}: {scheme}:// is not supported yet")).into())
+}
+
+/// A [`BlockCache`] over `backend` with its first block fetched (readahead
+/// of the superblock and the metadata usually written next to it).
+pub fn cached(backend: Backend, options: &Options) -> Result<RemoteStorage, Error> {
+    let cache = BlockCache::new(backend, options.cache.clone());
+    let first = cache.config().block_size;
+    cache
+        .prefetch(0, first)
+        .map_err(|e| Error::Hdf5(clawhdf5::Error::Format(e)))?;
+    Ok(cache)
+}