osobhandClaude Opus 5.5
db2554dd81
Range-read milestone M3, first half: a new crate with the block cache the
design makes mandatory for remote files and an HTTP backend, so
open_url("http://...") gives a clawhdf5::File over File::open_storage.
BlockCache wraps any Storage: aligned blocks (1 MiB by default, the size
docs/design/range-reads.md section 2 measured), LRU with a byte budget,
the missing blocks of one read_at/read_ranges fetched with one backend
read_ranges call as runs of consecutive blocks (a one-block gap filled to
merge runs, each request at most 8 MiB), and reads that miss more than
half the budget not kept. Thread-safe without holding the lock across a
fetch: a block being fetched is in flight, a second reader waits for it
instead of fetching it again, and a failed fetch fails its waiters and is
not cached. A backend holding the file in memory passes through.
HttpStorage (ureq, no TLS by default; `https` adds rustls with ring):
opening is one ranged GET of the first block, whose Content-Range gives
the length (the cache keeps the bytes). The file is pinned by a strong
ETag (If-Match), else Last-Modified (If-Unmodified-Since), and its length,
checked on every response: a change is RemoteError::FileChanged, never
mixed data. A server that ignores Range is refused without reading the
body unless a full download is allowed. Connection errors, timeouts,
408/429/5xx and short bodies are retried with exponential backoff;
Accept-Encoding: identity, and an encoded body is refused. read_ranges
fetches its ranges in parallel.
Tests (a std-only HTTP/1.1 server in tests/common/server.rs, also the
range_server example): every fixture read over HTTP gives File::open's
transcript (CLAWHDF5_REMOTE_CORPUS adds the conformance corpus), with
request counts per file with and without the cache; an h5py-written file
against libhdf5's values; a multi-block file fetched in whole blocks, each
once; a server ignoring Range; a file replaced mid-read (ETag,
Last-Modified, length only); truncated bodies and 503s (retried, then an
error, never cached); a slow server with 8 concurrent readers (no block
fetched twice); bad URLs, 404, encoded bodies, non-HDF5 data. The cache
has unit tests for coalescing, splitting, LRU order, large reads,
failures and concurrent in-flight dedup.
ci-test.sh: clawhdf5-remote joins the no-C default-build check, and its
https feature is linted.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
57 lines
2.0 KiB
Rust
57 lines
2.0 KiB
Rust
//! Serve the HDF5 files of a directory over HTTP with range support — the
|
|
//! server the tests use — to try `clawhdf5_remote` and `h5rs` on URLs.
|
|
//!
|
|
//! ```text
|
|
//! cargo run -p clawhdf5-remote --example range_server -- DIR [127.0.0.1:8000]
|
|
//! ```
|
|
//!
|
|
//! Every file under `DIR` is served at its path relative to `DIR`. The
|
|
//! files are read into memory at start. Requests are logged to stderr.
|
|
|
|
#[path = "../tests/common/server.rs"]
|
|
mod server;
|
|
|
|
fn main() {
|
|
let mut args = std::env::args().skip(1);
|
|
let Some(dir) = args.next() else {
|
|
eprintln!("usage: range_server DIR [ADDR]");
|
|
std::process::exit(2);
|
|
};
|
|
let addr = args.next().unwrap_or_else(|| "127.0.0.1:8000".into());
|
|
let root = std::path::PathBuf::from(&dir);
|
|
let mut files = Vec::new();
|
|
let mut stack = vec![root.clone()];
|
|
while let Some(d) = stack.pop() {
|
|
let Ok(entries) = std::fs::read_dir(&d) else {
|
|
continue;
|
|
};
|
|
for e in entries.flatten() {
|
|
let p = e.path();
|
|
if p.is_dir() {
|
|
stack.push(p);
|
|
} else if let (Ok(rel), Ok(bytes)) = (p.strip_prefix(&root), std::fs::read(&p)) {
|
|
let url = format!("/{}", rel.to_string_lossy().replace('\\', "/"));
|
|
files.push((url, bytes));
|
|
}
|
|
}
|
|
}
|
|
files.sort();
|
|
let server = server::Server::bind(&addr, files.clone());
|
|
for (path, bytes) in &files {
|
|
eprintln!("{} ({} bytes)", server.url(path), bytes.len());
|
|
}
|
|
eprintln!("serving {} files on http://{}", files.len(), server.addr);
|
|
let mut logged = 0;
|
|
loop {
|
|
std::thread::sleep(std::time::Duration::from_millis(200));
|
|
let log = server.log();
|
|
for (path, range) in &log[logged.min(log.len())..] {
|
|
match range {
|
|
Some((a, b)) => eprintln!("GET {path} bytes={a}-{b}"),
|
|
None => eprintln!("GET {path} (whole file)"),
|
|
}
|
|
}
|
|
logged = log.len();
|
|
}
|
|
}
|