clawhdf5-remote: block cache and HTTP range reads (open_url)
Range-read milestone M3, first half: a new crate with the block cache the
design makes mandatory for remote files and an HTTP backend, so
open_url("http://...") gives a clawhdf5::File over File::open_storage.
BlockCache wraps any Storage: aligned blocks (1 MiB by default, the size
docs/design/range-reads.md section 2 measured), LRU with a byte budget,
the missing blocks of one read_at/read_ranges fetched with one backend
read_ranges call as runs of consecutive blocks (a one-block gap filled to
merge runs, each request at most 8 MiB), and reads that miss more than
half the budget not kept. Thread-safe without holding the lock across a
fetch: a block being fetched is in flight, a second reader waits for it
instead of fetching it again, and a failed fetch fails its waiters and is
not cached. A backend holding the file in memory passes through.
HttpStorage (ureq, no TLS by default; `https` adds rustls with ring):
opening is one ranged GET of the first block, whose Content-Range gives
the length (the cache keeps the bytes). The file is pinned by a strong
ETag (If-Match), else Last-Modified (If-Unmodified-Since), and its length,
checked on every response: a change is RemoteError::FileChanged, never
mixed data. A server that ignores Range is refused without reading the
body unless a full download is allowed. Connection errors, timeouts,
408/429/5xx and short bodies are retried with exponential backoff;
Accept-Encoding: identity, and an encoded body is refused. read_ranges
fetches its ranges in parallel.
Tests (a std-only HTTP/1.1 server in tests/common/server.rs, also the
range_server example): every fixture read over HTTP gives File::open's
transcript (CLAWHDF5_REMOTE_CORPUS adds the conformance corpus), with
request counts per file with and without the cache; an h5py-written file
against libhdf5's values; a multi-block file fetched in whole blocks, each
once; a server ignoring Range; a file replaced mid-read (ETag,
Last-Modified, length only); truncated bodies and 503s (retried, then an
error, never cached); a slow server with 8 concurrent readers (no block
fetched twice); bad URLs, 404, encoded bodies, non-HDF5 data. The cache
has unit tests for coalescing, splitting, LRU order, large reads,
failures and concurrent in-flight dedup.
ci-test.sh: clawhdf5-remote joins the no-C default-build check, and its
https feature is linted.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
@@ -0,0 +1,565 @@
|
||||
//! HTTP(S) range requests: [`HttpStorage`].
|
||||
//!
|
||||
//! Every read is a `GET` with a `Range: bytes=a-b` header, answered `206
|
||||
//! Partial Content`. The file is pinned when it is opened:
|
||||
//!
|
||||
//! - its length comes from the `Content-Range` of the first request (which
|
||||
//! also fetches the first block, so opening costs one request);
|
||||
//! - a strong `ETag` is sent back as `If-Match` on every later request, and
|
||||
//! compared with the `ETag` of every response; without one, `Last-Modified`
|
||||
//! is sent as `If-Unmodified-Since` and compared; the length in every
|
||||
//! `Content-Range` must stay the same. A file that changes while it is
|
||||
//! open is [`RemoteError::FileChanged`], never a mix of old and new bytes.
|
||||
//! (A server that sends neither validator cannot be checked beyond the
|
||||
//! length; [`HttpOptions::require_validator`] refuses such servers.)
|
||||
//! - a server that ignores `Range` and answers `200` with the whole file is
|
||||
//! refused with [`RemoteError::RangeNotSupported`], unless
|
||||
//! [`HttpOptions::allow_full_download`] is set: then the file is
|
||||
//! downloaded once, at open, and read from memory.
|
||||
//!
|
||||
//! Transient failures — connection errors, timeouts, `408`/`429`/`5xx`, and
|
||||
//! a body shorter or longer than its `Content-Range` — are retried with
|
||||
//! exponential backoff. Responses are requested with
|
||||
//! `Accept-Encoding: identity`, since a compressed body cannot be a byte
|
||||
//! range of the file.
|
||||
//!
|
||||
//! `HttpStorage` itself does not cache: each `read_at` is one request. Read
|
||||
//! it through [`BlockCache`](crate::BlockCache) (which [`open_url`](crate::open_url)
|
||||
//! does); its `read_ranges` fetches the ranges of one call in parallel.
|
||||
|
||||
use std::borrow::Cow;
|
||||
use std::io::Read;
|
||||
use std::ops::Range;
|
||||
use std::sync::atomic::{AtomicU64, AtomicUsize, Ordering};
|
||||
use std::time::Duration;
|
||||
|
||||
use clawhdf5_format::error::FormatError;
|
||||
use clawhdf5_format::storage::Storage;
|
||||
|
||||
use crate::error::RemoteError;
|
||||
|
||||
/// Settings of an [`HttpStorage`].
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct HttpOptions {
|
||||
/// Retries of a request that failed transiently (so up to `retries + 1`
|
||||
/// attempts).
|
||||
pub retries: u32,
|
||||
/// Delay before the first retry; doubled for each further one.
|
||||
pub backoff: Duration,
|
||||
/// Timeout of one request, from connecting to the end of the body.
|
||||
pub timeout: Duration,
|
||||
/// Requests of one `read_ranges` call in flight at once.
|
||||
pub max_parallel: usize,
|
||||
/// Bytes fetched by the first request, from offset 0 (the superblock and
|
||||
/// usually the root group's metadata); at least 1.
|
||||
pub first_request: u64,
|
||||
/// When the server ignores `Range` (answers `200`), download the whole
|
||||
/// file once and read it from memory, instead of failing.
|
||||
pub allow_full_download: bool,
|
||||
/// Largest file [`allow_full_download`](Self::allow_full_download) will
|
||||
/// download.
|
||||
pub max_full_download: u64,
|
||||
/// Refuse a server that sends neither a strong `ETag` nor
|
||||
/// `Last-Modified`, since a change of the file could then go unnoticed
|
||||
/// (only its length is checked).
|
||||
pub require_validator: bool,
|
||||
/// Extra headers sent with every request (for example
|
||||
/// `Authorization`).
|
||||
pub headers: Vec<(String, String)>,
|
||||
}
|
||||
|
||||
impl Default for HttpOptions {
|
||||
fn default() -> Self {
|
||||
HttpOptions {
|
||||
retries: 3,
|
||||
backoff: Duration::from_millis(200),
|
||||
timeout: Duration::from_secs(60),
|
||||
max_parallel: 8,
|
||||
first_request: crate::cache::DEFAULT_BLOCK_SIZE,
|
||||
allow_full_download: false,
|
||||
max_full_download: 1 << 30,
|
||||
require_validator: false,
|
||||
headers: Vec::new(),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Requests and bytes an [`HttpStorage`] has used.
|
||||
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
|
||||
pub struct HttpStats {
|
||||
/// HTTP requests sent (retries included).
|
||||
pub requests: u64,
|
||||
/// Requests that were retries.
|
||||
pub retries: u64,
|
||||
/// Response body bytes received.
|
||||
pub bytes: u64,
|
||||
}
|
||||
|
||||
/// How the file is pinned.
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
enum Validator {
|
||||
ETag(String),
|
||||
LastModified(String),
|
||||
None,
|
||||
}
|
||||
|
||||
/// An HTTP(S) file read by range requests. See the [module
|
||||
/// documentation](self).
|
||||
pub struct HttpStorage {
|
||||
agent: ureq::Agent,
|
||||
url: String,
|
||||
len: u64,
|
||||
validator: Validator,
|
||||
options: HttpOptions,
|
||||
/// The whole file, when the server ignores ranges and a full download
|
||||
/// was allowed.
|
||||
full: Option<Vec<u8>>,
|
||||
requests: AtomicU64,
|
||||
retries: AtomicU64,
|
||||
bytes: AtomicU64,
|
||||
}
|
||||
|
||||
impl std::fmt::Debug for HttpStorage {
|
||||
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||
f.debug_struct("HttpStorage")
|
||||
.field("url", &self.url)
|
||||
.field("len", &self.len)
|
||||
.field("validator", &self.validator)
|
||||
.field("full_download", &self.full.is_some())
|
||||
.finish()
|
||||
}
|
||||
}
|
||||
|
||||
/// A parsed `Content-Range: bytes a-b/total`.
|
||||
fn content_range(v: &str) -> Option<(u64, u64, Option<u64>)> {
|
||||
let rest = v.trim().strip_prefix("bytes")?.trim_start();
|
||||
let (span, total) = rest.split_once('/')?;
|
||||
let (a, b) = span.trim().split_once('-')?;
|
||||
let a: u64 = a.trim().parse().ok()?;
|
||||
let b: u64 = b.trim().parse().ok()?;
|
||||
if b < a {
|
||||
return None;
|
||||
}
|
||||
let total = match total.trim() {
|
||||
"*" => None,
|
||||
t => Some(t.parse().ok()?),
|
||||
};
|
||||
Some((a, b, total))
|
||||
}
|
||||
|
||||
fn header<'a>(resp: &'a ureq::http::Response<ureq::Body>, name: &str) -> Option<&'a str> {
|
||||
resp.headers().get(name).and_then(|v| v.to_str().ok())
|
||||
}
|
||||
|
||||
/// A body with a `Content-Encoding` is not a byte range of the file.
|
||||
fn check_identity(url: &str, resp: &ureq::http::Response<ureq::Body>) -> Result<(), RemoteError> {
|
||||
match header(resp, "content-encoding") {
|
||||
Some(enc) if !enc.trim().eq_ignore_ascii_case("identity") => {
|
||||
Err(RemoteError::Usage(format!(
|
||||
"{url}: the server sent a {enc}-encoded body despite Accept-Encoding: identity"
|
||||
)))
|
||||
}
|
||||
_ => Ok(()),
|
||||
}
|
||||
}
|
||||
|
||||
fn transport(e: ureq::Error) -> RemoteError {
|
||||
match e {
|
||||
ureq::Error::StatusCode(code) => RemoteError::Status {
|
||||
code,
|
||||
what: "request".into(),
|
||||
},
|
||||
ureq::Error::BadUri(s) => RemoteError::InvalidUrl(s),
|
||||
other => RemoteError::Transport(other.to_string()),
|
||||
}
|
||||
}
|
||||
|
||||
impl HttpStorage {
|
||||
/// Open `url` (`http://`, or `https://` with the `https` feature):
|
||||
/// one ranged `GET` of the first [`HttpOptions::first_request`] bytes,
|
||||
/// which gives the file's length and validators. Returns the storage
|
||||
/// and the bytes that request fetched (the file's start), for a
|
||||
/// [`BlockCache`](crate::BlockCache) to keep.
|
||||
pub fn open(url: &str, options: HttpOptions) -> Result<(HttpStorage, Vec<u8>), RemoteError> {
|
||||
let lower = url.to_ascii_lowercase();
|
||||
if lower.starts_with("https://") {
|
||||
if !cfg!(feature = "https") {
|
||||
return Err(RemoteError::UnsupportedScheme(format!(
|
||||
"{url}: https:// needs the `https` feature of clawhdf5-remote"
|
||||
)));
|
||||
}
|
||||
} else if !lower.starts_with("http://") {
|
||||
return Err(RemoteError::UnsupportedScheme(url.to_string()));
|
||||
}
|
||||
let config = ureq::Agent::config_builder()
|
||||
.http_status_as_error(false)
|
||||
.timeout_global(Some(options.timeout))
|
||||
.build();
|
||||
let mut storage = HttpStorage {
|
||||
agent: ureq::Agent::new_with_config(config),
|
||||
url: url.to_string(),
|
||||
len: 0,
|
||||
validator: Validator::None,
|
||||
options,
|
||||
full: None,
|
||||
requests: AtomicU64::new(0),
|
||||
retries: AtomicU64::new(0),
|
||||
bytes: AtomicU64::new(0),
|
||||
};
|
||||
let first = storage.with_retries(|| storage.probe())?;
|
||||
let (len, validator, bytes, full) = first;
|
||||
storage.len = len;
|
||||
storage.validator = validator;
|
||||
if full {
|
||||
storage.full = Some(bytes);
|
||||
return Ok((storage, Vec::new()));
|
||||
}
|
||||
if storage.options.require_validator && storage.validator == Validator::None {
|
||||
return Err(RemoteError::Usage(format!(
|
||||
"{url}: the server sends neither a strong ETag nor Last-Modified, so a change \
|
||||
of the file could not be detected (HttpOptions::require_validator)"
|
||||
)));
|
||||
}
|
||||
Ok((storage, bytes))
|
||||
}
|
||||
|
||||
/// The URL.
|
||||
pub fn url(&self) -> &str {
|
||||
&self.url
|
||||
}
|
||||
|
||||
/// Whether the file was downloaded whole, because the server does not
|
||||
/// support ranges and [`HttpOptions::allow_full_download`] was set.
|
||||
pub fn is_full_download(&self) -> bool {
|
||||
self.full.is_some()
|
||||
}
|
||||
|
||||
/// The `ETag` the file is pinned to, if the server sent a strong one.
|
||||
pub fn etag(&self) -> Option<&str> {
|
||||
match &self.validator {
|
||||
Validator::ETag(e) => Some(e),
|
||||
_ => None,
|
||||
}
|
||||
}
|
||||
|
||||
/// Requests and bytes so far.
|
||||
pub fn stats(&self) -> HttpStats {
|
||||
HttpStats {
|
||||
requests: self.requests.load(Ordering::Relaxed),
|
||||
retries: self.retries.load(Ordering::Relaxed),
|
||||
bytes: self.bytes.load(Ordering::Relaxed),
|
||||
}
|
||||
}
|
||||
|
||||
fn with_retries<T>(
|
||||
&self,
|
||||
mut attempt: impl FnMut() -> Result<T, RemoteError>,
|
||||
) -> Result<T, RemoteError> {
|
||||
let mut delay = self.options.backoff;
|
||||
let mut n = 0;
|
||||
loop {
|
||||
match attempt() {
|
||||
Ok(v) => return Ok(v),
|
||||
Err(e) if e.is_transient() && n < self.options.retries => {
|
||||
n += 1;
|
||||
self.retries.fetch_add(1, Ordering::Relaxed);
|
||||
std::thread::sleep(delay);
|
||||
delay = delay.saturating_mul(2);
|
||||
}
|
||||
Err(e) => return Err(e),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn request(
|
||||
&self,
|
||||
range: Option<(u64, u64)>,
|
||||
) -> ureq::RequestBuilder<ureq::typestate::WithoutBody> {
|
||||
let mut req = self
|
||||
.agent
|
||||
.get(&self.url)
|
||||
.header("Accept-Encoding", "identity");
|
||||
if let Some((a, b)) = range {
|
||||
req = req.header("Range", format!("bytes={a}-{b}"));
|
||||
}
|
||||
match &self.validator {
|
||||
Validator::ETag(e) => req = req.header("If-Match", e),
|
||||
Validator::LastModified(t) => req = req.header("If-Unmodified-Since", t),
|
||||
Validator::None => {}
|
||||
}
|
||||
for (k, v) in &self.options.headers {
|
||||
req = req.header(k, v);
|
||||
}
|
||||
req
|
||||
}
|
||||
|
||||
/// Read a body of exactly `want` bytes (or up to `limit` when `want` is
|
||||
/// unknown).
|
||||
fn body(
|
||||
&self,
|
||||
resp: ureq::http::Response<ureq::Body>,
|
||||
want: Option<u64>,
|
||||
limit: u64,
|
||||
) -> Result<Vec<u8>, RemoteError> {
|
||||
let cap = want.unwrap_or(limit);
|
||||
let mut buf = Vec::with_capacity(usize::try_from(cap.min(64 << 20)).unwrap_or(0));
|
||||
let reader = resp.into_body().into_reader();
|
||||
let got = reader
|
||||
.take(cap.saturating_add(1))
|
||||
.read_to_end(&mut buf)
|
||||
.map_err(|e| RemoteError::Transport(format!("{}: reading the body: {e}", self.url)));
|
||||
self.bytes.fetch_add(buf.len() as u64, Ordering::Relaxed);
|
||||
got?;
|
||||
match want {
|
||||
Some(n) if buf.len() as u64 != n => Err(RemoteError::BadResponse(format!(
|
||||
"{}: body of {} bytes, expected {n}",
|
||||
self.url,
|
||||
buf.len()
|
||||
))),
|
||||
None if buf.len() as u64 > limit => Err(RemoteError::Usage(format!(
|
||||
"{}: the file is larger than HttpOptions::max_full_download ({limit} bytes)",
|
||||
self.url
|
||||
))),
|
||||
_ => Ok(buf),
|
||||
}
|
||||
}
|
||||
|
||||
/// The first request: length, validators and the file's first bytes.
|
||||
/// The last field is true when the server ignored the range and sent
|
||||
/// the whole file (only kept when a full download is allowed).
|
||||
fn probe(&self) -> Result<(u64, Validator, Vec<u8>, bool), RemoteError> {
|
||||
let n = self.options.first_request.max(1);
|
||||
self.requests.fetch_add(1, Ordering::Relaxed);
|
||||
let resp = self.request(Some((0, n - 1))).call().map_err(transport)?;
|
||||
let status = resp.status().as_u16();
|
||||
check_identity(&self.url, &resp)?;
|
||||
let validator = match (header(&resp, "etag"), header(&resp, "last-modified")) {
|
||||
(Some(e), _) if !e.starts_with("W/") => Validator::ETag(e.to_string()),
|
||||
(_, Some(t)) => Validator::LastModified(t.to_string()),
|
||||
_ => Validator::None,
|
||||
};
|
||||
match status {
|
||||
206 => {
|
||||
let cr = header(&resp, "content-range").ok_or_else(|| {
|
||||
RemoteError::BadResponse(format!("{}: 206 without Content-Range", self.url))
|
||||
})?;
|
||||
let (a, b, total) = content_range(cr).ok_or_else(|| {
|
||||
RemoteError::BadResponse(format!("{}: bad Content-Range {cr:?}", self.url))
|
||||
})?;
|
||||
let total = total.ok_or_else(|| {
|
||||
RemoteError::BadResponse(format!(
|
||||
"{}: the server does not report the file's length (Content-Range {cr:?})",
|
||||
self.url
|
||||
))
|
||||
})?;
|
||||
if a != 0 || b >= total || b > n - 1 {
|
||||
return Err(RemoteError::BadResponse(format!(
|
||||
"{}: asked for bytes 0-{}, got Content-Range {cr:?}",
|
||||
self.url,
|
||||
n - 1
|
||||
)));
|
||||
}
|
||||
let bytes = self.body(resp, Some(b - a + 1), 0)?;
|
||||
Ok((total, validator, bytes, false))
|
||||
}
|
||||
200 => {
|
||||
if !self.options.allow_full_download {
|
||||
return Err(RemoteError::RangeNotSupported(format!(
|
||||
"{} answered a range request with the whole file (status 200); set \
|
||||
HttpOptions::allow_full_download to download it",
|
||||
self.url
|
||||
)));
|
||||
}
|
||||
let want = header(&resp, "content-length").and_then(|v| v.trim().parse().ok());
|
||||
if want.is_some_and(|w: u64| w > self.options.max_full_download) {
|
||||
return Err(RemoteError::Usage(format!(
|
||||
"{}: the file is larger than HttpOptions::max_full_download ({} bytes)",
|
||||
self.url, self.options.max_full_download
|
||||
)));
|
||||
}
|
||||
let bytes = self.body(resp, want, self.options.max_full_download)?;
|
||||
Ok((bytes.len() as u64, validator, bytes, true))
|
||||
}
|
||||
416 => Err(RemoteError::Usage(format!(
|
||||
"{}: status 416 for the first bytes (an empty file?)",
|
||||
self.url
|
||||
))),
|
||||
code => Err(RemoteError::Status {
|
||||
code,
|
||||
what: self.url.clone(),
|
||||
}),
|
||||
}
|
||||
}
|
||||
|
||||
/// One request for `[start, end)` (inside the file, non-empty).
|
||||
fn fetch_once(&self, start: u64, end: u64) -> Result<Vec<u8>, RemoteError> {
|
||||
self.requests.fetch_add(1, Ordering::Relaxed);
|
||||
let resp = self
|
||||
.request(Some((start, end - 1)))
|
||||
.call()
|
||||
.map_err(transport)?;
|
||||
let changed = |why: String| RemoteError::FileChanged(format!("{}: {why}", self.url));
|
||||
check_identity(&self.url, &resp)?;
|
||||
match resp.status().as_u16() {
|
||||
206 => {}
|
||||
200 => {
|
||||
return Err(RemoteError::RangeNotSupported(format!(
|
||||
"{} answered a range request with the whole file (status 200)",
|
||||
self.url
|
||||
)));
|
||||
}
|
||||
412 => {
|
||||
return Err(changed(
|
||||
"If-Match/If-Unmodified-Since failed (status 412)".into(),
|
||||
));
|
||||
}
|
||||
416 => return Err(changed("range no longer satisfiable (status 416)".into())),
|
||||
code => {
|
||||
return Err(RemoteError::Status {
|
||||
code,
|
||||
what: format!("{} bytes {start}-{}", self.url, end - 1),
|
||||
});
|
||||
}
|
||||
}
|
||||
match &self.validator {
|
||||
Validator::ETag(e) => {
|
||||
if let Some(got) = header(&resp, "etag")
|
||||
&& got != e
|
||||
{
|
||||
return Err(changed(format!("ETag {got} instead of {e}")));
|
||||
}
|
||||
}
|
||||
Validator::LastModified(t) => {
|
||||
if let Some(got) = header(&resp, "last-modified")
|
||||
&& got != t
|
||||
{
|
||||
return Err(changed(format!("Last-Modified {got} instead of {t}")));
|
||||
}
|
||||
}
|
||||
Validator::None => {}
|
||||
}
|
||||
let cr = header(&resp, "content-range").ok_or_else(|| {
|
||||
RemoteError::BadResponse(format!("{}: 206 without Content-Range", self.url))
|
||||
})?;
|
||||
let (a, b, total) = content_range(cr).ok_or_else(|| {
|
||||
RemoteError::BadResponse(format!("{}: bad Content-Range {cr:?}", self.url))
|
||||
})?;
|
||||
if let Some(total) = total
|
||||
&& total != self.len
|
||||
{
|
||||
return Err(changed(format!("length {total} instead of {}", self.len)));
|
||||
}
|
||||
if a != start || b != end - 1 {
|
||||
return Err(RemoteError::BadResponse(format!(
|
||||
"{}: asked for bytes {start}-{}, got Content-Range {cr:?}",
|
||||
self.url,
|
||||
end - 1
|
||||
)));
|
||||
}
|
||||
self.body(resp, Some(end - start), 0)
|
||||
}
|
||||
|
||||
fn fetch(&self, start: u64, end: u64) -> Result<Vec<u8>, RemoteError> {
|
||||
self.with_retries(|| self.fetch_once(start, end))
|
||||
}
|
||||
|
||||
/// `[offset, offset + len)` clamped to the file, or `None` if empty.
|
||||
fn clamp(&self, offset: u64, len: u64) -> Option<(u64, u64)> {
|
||||
let end = offset.saturating_add(len).min(self.len);
|
||||
(offset < end).then_some((offset, end))
|
||||
}
|
||||
}
|
||||
|
||||
impl Storage for HttpStorage {
|
||||
fn read_at(&self, offset: u64, len: usize) -> Result<Cow<'_, [u8]>, FormatError> {
|
||||
if let Some(all) = &self.full {
|
||||
return all.as_slice().read_at(offset, len);
|
||||
}
|
||||
match self.clamp(offset, len as u64) {
|
||||
None => Ok(Cow::Owned(Vec::new())),
|
||||
Some((a, b)) => Ok(Cow::Owned(self.fetch(a, b)?)),
|
||||
}
|
||||
}
|
||||
|
||||
fn len(&self) -> u64 {
|
||||
self.len
|
||||
}
|
||||
|
||||
fn read_ranges(&self, ranges: &[Range<u64>]) -> Result<Vec<Cow<'_, [u8]>>, FormatError> {
|
||||
if let Some(all) = &self.full {
|
||||
return all.as_slice().read_ranges(ranges);
|
||||
}
|
||||
let parallel = self.options.max_parallel.clamp(1, ranges.len().max(1));
|
||||
if parallel <= 1 {
|
||||
return ranges
|
||||
.iter()
|
||||
.map(|r| self.read_at(r.start, (r.end.saturating_sub(r.start)) as usize))
|
||||
.collect();
|
||||
}
|
||||
let next = AtomicUsize::new(0);
|
||||
let failed = std::sync::atomic::AtomicBool::new(false);
|
||||
type Slot = Option<Result<Vec<u8>, RemoteError>>;
|
||||
let results: std::sync::Mutex<Vec<Slot>> =
|
||||
std::sync::Mutex::new((0..ranges.len()).map(|_| None).collect());
|
||||
std::thread::scope(|s| {
|
||||
for _ in 0..parallel {
|
||||
s.spawn(|| {
|
||||
loop {
|
||||
let i = next.fetch_add(1, Ordering::Relaxed);
|
||||
if i >= ranges.len() || failed.load(Ordering::Relaxed) {
|
||||
break;
|
||||
}
|
||||
let r = &ranges[i];
|
||||
let got = match self.clamp(r.start, r.end.saturating_sub(r.start)) {
|
||||
None => Ok(Vec::new()),
|
||||
Some((a, b)) => self.fetch(a, b),
|
||||
};
|
||||
if got.is_err() {
|
||||
failed.store(true, Ordering::Relaxed);
|
||||
}
|
||||
results
|
||||
.lock()
|
||||
.unwrap_or_else(std::sync::PoisonError::into_inner)[i] = Some(got);
|
||||
}
|
||||
});
|
||||
}
|
||||
});
|
||||
let results = results
|
||||
.into_inner()
|
||||
.unwrap_or_else(std::sync::PoisonError::into_inner);
|
||||
let mut out = Vec::with_capacity(ranges.len());
|
||||
for r in results {
|
||||
match r {
|
||||
Some(Ok(v)) => out.push(Cow::Owned(v)),
|
||||
Some(Err(e)) => return Err(e.into()),
|
||||
// Not fetched because another range failed first.
|
||||
None => continue,
|
||||
}
|
||||
}
|
||||
if out.len() != ranges.len() {
|
||||
return Err(FormatError::Storage(format!(
|
||||
"{}: a parallel range read failed",
|
||||
self.url
|
||||
)));
|
||||
}
|
||||
Ok(out)
|
||||
}
|
||||
|
||||
fn as_contiguous(&self) -> Option<&[u8]> {
|
||||
self.full.as_deref()
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::content_range;
|
||||
|
||||
#[test]
|
||||
fn content_range_parses() {
|
||||
assert_eq!(content_range("bytes 0-99/1000"), Some((0, 99, Some(1000))));
|
||||
assert_eq!(content_range("bytes 5-5/*"), Some((5, 5, None)));
|
||||
assert_eq!(content_range("bytes 9-5/10"), None);
|
||||
assert_eq!(content_range("items 0-1/2"), None);
|
||||
assert_eq!(content_range("bytes */1000"), None);
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user