clawhdf5-remote: block cache and HTTP range reads (open_url)

Range-read milestone M3, first half: a new crate with the block cache the
design makes mandatory for remote files and an HTTP backend, so
open_url("http://...") gives a clawhdf5::File over File::open_storage.

BlockCache wraps any Storage: aligned blocks (1 MiB by default, the size
docs/design/range-reads.md section 2 measured), LRU with a byte budget,
the missing blocks of one read_at/read_ranges fetched with one backend
read_ranges call as runs of consecutive blocks (a one-block gap filled to
merge runs, each request at most 8 MiB), and reads that miss more than
half the budget not kept. Thread-safe without holding the lock across a
fetch: a block being fetched is in flight, a second reader waits for it
instead of fetching it again, and a failed fetch fails its waiters and is
not cached. A backend holding the file in memory passes through.

HttpStorage (ureq, no TLS by default; `https` adds rustls with ring):
opening is one ranged GET of the first block, whose Content-Range gives
the length (the cache keeps the bytes). The file is pinned by a strong
ETag (If-Match), else Last-Modified (If-Unmodified-Since), and its length,
checked on every response: a change is RemoteError::FileChanged, never
mixed data. A server that ignores Range is refused without reading the
body unless a full download is allowed. Connection errors, timeouts,
408/429/5xx and short bodies are retried with exponential backoff;
Accept-Encoding: identity, and an encoded body is refused. read_ranges
fetches its ranges in parallel.

Tests (a std-only HTTP/1.1 server in tests/common/server.rs, also the
range_server example): every fixture read over HTTP gives File::open's
transcript (CLAWHDF5_REMOTE_CORPUS adds the conformance corpus), with
request counts per file with and without the cache; an h5py-written file
against libhdf5's values; a multi-block file fetched in whole blocks, each
once; a server ignoring Range; a file replaced mid-read (ETag,
Last-Modified, length only); truncated bodies and 503s (retried, then an
error, never cached); a slow server with 8 concurrent readers (no block
fetched twice); bad URLs, 404, encoded bodies, non-HDF5 data. The cache
has unit tests for coalescing, splitting, LRU order, large reads,
failures and concurrent in-flight dedup.

ci-test.sh: clawhdf5-remote joins the no-C default-build check, and its
https feature is linted.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

crates/clawhdf5-remote/src/error.rs

@@ -0,0 +1,125 @@
+//! Errors of the remote backends.
+
+use clawhdf5_format::error::FormatError;
+
+/// Why a remote file could not be opened or read.
+///
+/// Inside a [`clawhdf5::File`] read these arrive as
+/// `clawhdf5::Error::Format(FormatError::Storage(message))`, the message
+/// being this error's `Display`.
+#[derive(Debug, Clone, PartialEq, Eq)]
+#[non_exhaustive]
+pub enum RemoteError {
+    /// The URL is malformed.
+    InvalidUrl(String),
+    /// The URL's scheme is not supported by this build (for example
+    /// `s3://` without the `s3` feature, or `https://` without `https`).
+    UnsupportedScheme(String),
+    /// The server answered a range request with the whole file (status
+    /// 200), i.e. it does not support ranges, and a full download was not
+    /// allowed ([`HttpOptions::allow_full_download`](crate::HttpOptions)).
+    RangeNotSupported(String),
+    /// The file changed since it was opened (a different ETag,
+    /// Last-Modified or length, or a failed `If-Match` precondition).
+    /// Nothing read after the change is returned.
+    FileChanged(String),
+    /// The server answered with an unexpected status.
+    Status {
+        /// The HTTP status code.
+        code: u16,
+        /// What was being requested.
+        what: String,
+    },
+    /// A response did not carry what was asked for (a wrong
+    /// `Content-Range`, a body shorter or longer than announced), after
+    /// every retry.
+    BadResponse(String),
+    /// A network failure (connection, timeout, reset) after every retry.
+    Transport(String),
+    /// An error from the object store.
+    ObjectStore(String),
+    /// Called in a way the backend cannot serve (for example a blocking
+    /// read from inside an async runtime).
+    Usage(String),
+}
+
+impl RemoteError {
+    /// Whether retrying the same request may succeed.
+    #[cfg_attr(not(feature = "http"), allow(dead_code))]
+    pub(crate) fn is_transient(&self) -> bool {
+        match self {
+            RemoteError::Transport(_) | RemoteError::BadResponse(_) => true,
+            RemoteError::Status { code, .. } => {
+                matches!(code, 408 | 429 | 500 | 502 | 503 | 504)
+            }
+            _ => false,
+        }
+    }
+}
+
+impl std::fmt::Display for RemoteError {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        match self {
+            RemoteError::InvalidUrl(s) => write!(f, "invalid URL: {s}"),
+            RemoteError::UnsupportedScheme(s) => write!(f, "unsupported URL: {s}"),
+            RemoteError::RangeNotSupported(s) => {
+                write!(f, "the server does not support range requests: {s}")
+            }
+            RemoteError::FileChanged(s) => write!(f, "the remote file changed while open: {s}"),
+            RemoteError::Status { code, what } => write!(f, "HTTP status {code} for {what}"),
+            RemoteError::BadResponse(s) => write!(f, "bad response: {s}"),
+            RemoteError::Transport(s) => write!(f, "network error: {s}"),
+            RemoteError::ObjectStore(s) => write!(f, "object store: {s}"),
+            RemoteError::Usage(s) => write!(f, "{s}"),
+        }
+    }
+}
+
+impl std::error::Error for RemoteError {}
+
+impl From<RemoteError> for FormatError {
+    fn from(e: RemoteError) -> Self {
+        FormatError::Storage(e.to_string())
+    }
+}
+
+/// An error of [`open_url`](crate::open_url): the remote side, or the file
+/// itself.
+#[derive(Debug)]
+#[non_exhaustive]
+pub enum Error {
+    /// Reaching or reading the remote file failed.
+    Remote(RemoteError),
+    /// The bytes were read, but they are not an HDF5 file clawhdf5 can open.
+    Hdf5(clawhdf5::Error),
+}
+
+impl std::fmt::Display for Error {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        match self {
+            Error::Remote(e) => e.fmt(f),
+            Error::Hdf5(e) => e.fmt(f),
+        }
+    }
+}
+
+impl std::error::Error for Error {
+    fn source(&self) -> Option<&(dyn std::error::Error + 'static)> {
+        match self {
+            Error::Remote(e) => Some(e),
+            Error::Hdf5(e) => Some(e),
+        }
+    }
+}
+
+impl From<RemoteError> for Error {
+    fn from(e: RemoteError) -> Self {
+        Error::Remote(e)
+    }
+}
+
+impl From<clawhdf5::Error> for Error {
+    fn from(e: clawhdf5::Error) -> Self {
+        Error::Hdf5(e)
+    }
+}