From 910d81904c1ef5e48977a8095d1b6049967c9032 Mon Sep 17 00:00:00 2001 From: osobh Date: Sun, 27 Sep 2026 06:40:44 -0500 Subject: [PATCH] py: remote files (clawhdf5.File(url), File.open_url) through File::storage() The Python bindings could not open a remote file: they parsed through File::as_bytes() in eight places (path lookups, object headers, dataspaces, attributes, group listings, the global heap of variable-length data), which a storage-backed file does not have. - Every object of a File now shares one handle (src/handle.rs) that runs all file access, metadata included, with the GIL released and parses through File::storage() and the clawhdf5_format *_in functions. Local files take the same path (their storage is the mmap). - clawhdf5.File(url) opens any scheme://... through clawhdf5_remote::storage_for_url (read-only; another mode is a ValueError). File.open_url(url, **options) takes the cache and HTTP options (block_size, cache_size, headers, retries, timeout, allow_full_download, max_full_download, require_validator, max_redirects, max_parallel); File.remote_stats gives the block cache's counters. - Default build: plain HTTP only, no C. https (rustls/ring) and s3/gcs/azure (aws-lc-rs) are opt-in features of clawhdf5-py, and ci-test.sh's no-C check now covers the crate. - A failed storage read (network error, file changed on the server) is an OSError, never KeyError/ValueError and never data; `key in group` raises it instead of answering False. Tests: the read-vs-h5py suite runs locally and over HTTP (1 MiB and 1 KiB blocks) against a range-capable http.server in the test process (conftest.RangeServer); test_remote.py covers request counts, cache hits, a server without Range support, a changed file, a server that hangs up, 16 threads, and a spinning thread that keeps running while a read waits on 0.2 s requests. Co-Authored-By: Claude Opus 5.5 (1M context) --- CHANGELOG.md | 30 +++ README.md | 12 + crates/clawhdf5-py/Cargo.toml | 9 + crates/clawhdf5-py/README.md | 35 ++- crates/clawhdf5-py/src/attrs.rs | 42 +-- crates/clawhdf5-py/src/convert.rs | 70 +++-- crates/clawhdf5-py/src/dataset.rs | 144 +++++++---- crates/clawhdf5-py/src/file.rs | 206 ++++++++++++--- crates/clawhdf5-py/src/group.rs | 151 +++++------ crates/clawhdf5-py/src/handle.rs | 130 ++++++++++ crates/clawhdf5-py/src/lib.rs | 7 +- crates/clawhdf5-py/src/node.rs | 165 ++++++------ crates/clawhdf5-py/tests/conftest.py | 135 ++++++++++ crates/clawhdf5-py/tests/test_read_vs_h5py.py | 25 +- crates/clawhdf5-py/tests/test_remote.py | 244 ++++++++++++++++++ docs/design/range-reads.md | 18 +- docs/known-issues.md | 15 +- scripts/ci-test.sh | 7 +- 18 files changed, 1146 insertions(+), 299 deletions(-) create mode 100644 crates/clawhdf5-py/src/handle.rs create mode 100644 crates/clawhdf5-py/tests/test_remote.py diff --git a/CHANGELOG.md b/CHANGELOG.md index 664a831..6b153b2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,36 @@ ## Unreleased +### Python bindings: remote files (2026-09-27) +- **`clawhdf5.File(url)`** opens `http://` URLs (and `https://`, `s3://`, + `gs://`, `az://` in a wheel built with the `https`, `s3`, `gcs`, + `azure` features) through `clawhdf5-remote`'s `open_url`: range requests + through the block cache, the whole read API (groups, attributes, every + dataset type and index the local reader handles). A URL is any + `scheme://…`; a remote file is read-only (another mode is a + `ValueError`). **`clawhdf5.File.open_url(url, **options)`** takes + `block_size`, `cache_size`, `headers`, `retries`, `timeout`, + `allow_full_download`, `max_full_download`, `require_validator`, + `max_redirects` and `max_parallel`; `File.remote_stats` gives the block + cache's counters. The default wheel builds plain HTTP only (no C: rustls + needs ring, and the cloud clients aws-lc-rs), and `ci-test.sh`'s no-C + check now covers `clawhdf5-py`. +- **Every read parses through `File::storage()`** instead of + `File::as_bytes()` (path lookups, object headers, dataspaces, attributes, + group listings, variable-length data through the global heap), inside one + shared file handle that releases the GIL for all file access, not only + dataset reads: a read waiting on the network lets other Python threads + run. A failed read of the storage (a network error, a file changed on + the server) is an `OSError`, never a `KeyError`/`ValueError` and never + data; `key in group` raises it instead of answering `False`. +- Tests: the whole read-vs-h5py suite also runs over HTTP (default 1 MiB + blocks and 1 KiB blocks) against a range-capable `http.server` in the + test process, plus `tests/test_remote.py`: request counts of a small + read, cache hits, a server without `Range` support (refused, or a + whole download when allowed), a file changed on the server, a server + that hangs up, 16 threads on one remote file, and a thread that keeps + running while a read waits on 0.2 s requests. + ### Range reads, milestone M3: remote files (2026-09-26) - **New crate `clawhdf5-remote`.** `open_url("http://host/file.h5")` gives a `clawhdf5::File` (through `File::open_storage`) that reads the file by diff --git a/README.md b/README.md index 3301b09..dccee48 100644 --- a/README.md +++ b/README.md @@ -533,8 +533,20 @@ with clawhdf5.File("data.h5", "r") as f: records = f["table"] # compound -> numpy structured array ids = records["id"] # one field + +# A file on a web server: range requests through a block cache, nothing +# downloaded up front; the same read API. The GIL is released while waiting. +with clawhdf5.File("http://data.example.org/run42.h5") as f: + first = f["group/temperatures"][0] +f = clawhdf5.File.open_url("http://data.example.org/run42.h5", block_size=256 * 1024, + headers={"Authorization": "Bearer ..."}) ``` +The default build reads `http://` URLs only; build with +`maturin develop --release --features https` (rustls with ring, which +compiles C) for `https://`, and `--features s3` (or `gcs`, `azure`) for +object-store URLs. + Reads cover integers and IEEE floats of every width in either byte order, `bool`, enums, complex, fixed and variable-length strings, variable-length sequences, opaque, HDF5 array types and compounds; other types (references, diff --git a/crates/clawhdf5-py/Cargo.toml b/crates/clawhdf5-py/Cargo.toml index 3513a62..8246440 100644 --- a/crates/clawhdf5-py/Cargo.toml +++ b/crates/clawhdf5-py/Cargo.toml @@ -17,11 +17,20 @@ crate-type = ["cdylib", "rlib"] [dependencies] clawhdf5_rs = { path = "../clawhdf5", version = "2.7.0", package = "clawhdf5" } clawhdf5-format = { path = "../clawhdf5-format", version = "2.7.0" } +# Remote files (`clawhdf5.File(url)`): plain HTTP by default, which builds +# no C. HTTPS and the object stores are opt-in features below. +clawhdf5-remote = { path = "../clawhdf5-remote", version = "2.7.0" } pyo3 = "0.29" numpy = "0.29" [features] extension-module = ["pyo3/extension-module"] +# https:// URLs (rustls with ring, which compiles C and assembly). +https = ["clawhdf5-remote/https"] +# s3://, gs://, az:// URLs (object_store; its cloud clients build aws-lc-rs, C). +s3 = ["clawhdf5-remote/s3"] +gcs = ["clawhdf5-remote/gcs"] +azure = ["clawhdf5-remote/azure"] [package.metadata.docs.rs] features = [] diff --git a/crates/clawhdf5-py/README.md b/crates/clawhdf5-py/README.md index 95cae12..16e4169 100644 --- a/crates/clawhdf5-py/README.md +++ b/crates/clawhdf5-py/README.md @@ -61,6 +61,36 @@ with clawhdf5.File("data.h5", "r") as f: - Attributes return what h5py returns; `clawhdf5.Empty` stands for a null dataspace (h5py's `Empty`). +## Remote files + +A URL instead of a path reads the file where it is, through +`clawhdf5-remote`: HTTP range requests through a block cache (1 MiB blocks, +64 MiB budget by default), fetching only the blocks a read needs. The whole +read API works the same, and the GIL is released while waiting on the +network. + +```python +f = clawhdf5.File("http://host/data.h5") # default options +f = clawhdf5.File.open_url( + "http://host/data.h5", + block_size=256 * 1024, cache_size=128 << 20, # the block cache + headers={"Authorization": "Bearer ..."}, # sent to this origin only + retries=3, timeout=30.0, max_redirects=5, max_parallel=8, + allow_full_download=False, # a server without Range support: refuse + require_validator=False, # refuse servers without ETag/Last-Modified +) +f.remote_stats # {'requests': ..., 'bytes_fetched': ..., 'hits': ..., ...} +``` + +- The file is pinned when opened (ETag or Last-Modified, and length): if it + changes on the server, reads raise `OSError` instead of mixing versions. + Network failures are `OSError` too. +- Remote files are read-only. +- Schemes: the default build (no C) reads `http://`. `https://` needs + `maturin develop --release --features https` (rustls with ring, which + compiles C); `s3://`, `gs://` and `az://` need the `s3`, `gcs` and + `azure` features (credentials from the environment; aws-lc-rs, C). + ## Writing `clawhdf5.File(path, "w")` with `create_dataset(name, data=array, @@ -76,7 +106,10 @@ pytest crates/clawhdf5-py/tests ``` `tests/test_read_vs_h5py.py` compares every read with h5py on a file h5py -writes. `scripts/ci-test.sh` builds the wheel and runs these in CI. +writes, opened locally and over HTTP (an in-process range server, +`tests/conftest.py`); `tests/test_remote.py` checks remote reads (requests, +failures, the GIL). `scripts/ci-test.sh` builds the wheel and runs these +in CI. ## License diff --git a/crates/clawhdf5-py/src/attrs.rs b/crates/clawhdf5-py/src/attrs.rs index fd3f0b3..615214b 100644 --- a/crates/clawhdf5-py/src/attrs.rs +++ b/crates/clawhdf5-py/src/attrs.rs @@ -8,13 +8,14 @@ use pyo3::prelude::*; use pyo3::types::{PyList, PyTuple}; use crate::convert::{Converter, Elements, resolve_vl}; +use crate::handle::Handle; use crate::{OwnedAttrValue, PyEmpty, attr_value_to_py, node, py_to_attr_value}; /// Backing storage for attributes. enum AttrsInner { /// Attributes of an object in a file opened for reading, sorted by name. Read { - file: Arc, + handle: Arc, attrs: Vec, }, /// Writable attribute list shared with a parent (PyFile or PyGroup). @@ -36,10 +37,15 @@ pub struct PyAttrs { impl PyAttrs { /// The attributes of the object at `addr` (whose path is `path`) in a /// file opened for reading. - pub(crate) fn read(file: Arc, addr: u64, path: &str) -> PyResult { - let attrs = node::attributes(&file, addr, path)?; + pub(crate) fn read( + py: Python<'_>, + handle: Arc, + addr: u64, + path: &str, + ) -> PyResult { + let attrs = handle.with(py, |f| node::attributes(f, addr, path))?; Ok(Self { - inner: AttrsInner::Read { file, attrs }, + inner: AttrsInner::Read { handle, attrs }, }) } @@ -55,8 +61,8 @@ impl PyAttrs { impl PyAttrs { fn __getitem__(&self, py: Python<'_>, key: &str) -> PyResult> { match &self.inner { - AttrsInner::Read { file, attrs } => match attrs.iter().find(|a| a.name == key) { - Some(attr) => Ok(attr_to_py(py, file, attr)?.unbind()), + AttrsInner::Read { handle, attrs } => match attrs.iter().find(|a| a.name == key) { + Some(attr) => Ok(attr_to_py(py, handle, attr)?.unbind()), None => Err(PyKeyError::new_err(format!( "Can't open attribute (can't locate attribute: '{key}')" ))), @@ -146,9 +152,9 @@ impl PyAttrs { /// Return attribute values as a list. fn values(&self, py: Python<'_>) -> PyResult> { let vals: Vec> = match &self.inner { - AttrsInner::Read { file, attrs } => attrs + AttrsInner::Read { handle, attrs } => attrs .iter() - .map(|a| attr_to_py(py, file, a).map(Bound::unbind)) + .map(|a| attr_to_py(py, handle, a).map(Bound::unbind)) .collect::>()?, AttrsInner::Write(store) => store .lock() @@ -167,9 +173,9 @@ impl PyAttrs { /// Return attribute (key, value) pairs as a list of tuples. fn items(&self, py: Python<'_>) -> PyResult> { let pairs: Vec<(String, Py)> = match &self.inner { - AttrsInner::Read { file, attrs } => attrs + AttrsInner::Read { handle, attrs } => attrs .iter() - .map(|a| Ok((a.name.clone(), attr_to_py(py, file, a)?.unbind()))) + .map(|a| Ok((a.name.clone(), attr_to_py(py, handle, a)?.unbind()))) .collect::>()?, AttrsInner::Write(store) => store .lock() @@ -189,12 +195,11 @@ impl PyAttrs { /// An attribute's value as h5py returns it. fn attr_to_py<'py>( py: Python<'py>, - file: &clawhdf5_rs::File, + handle: &Handle, attr: &AttributeMessage, ) -> PyResult> { crate::no_panic(|| { - let sb = file.superblock(); - let conv = Converter::new(py, &attr.datatype, sb.offset_size) + let conv = Converter::new(py, &attr.datatype, handle.offset_size) .map_err(|e| prefix_err(py, &attr.name, e))?; if node::is_null(&attr.dataspace) { return Ok(PyEmpty::new(conv.dtype).into_pyobject(py)?.into_any()); @@ -216,12 +221,11 @@ fn attr_to_py<'py>( ))); } let raw = &attr.raw_data[..want]; - let file_data = file.as_bytes(); - let (osz, lsz, unit) = (sb.offset_size, sb.length_size, conv.vl_unit); - Elements::Vl( - py.detach(|| resolve_vl(file_data, raw, n, osz, lsz, unit)) - .map_err(|e| PyValueError::new_err(format!("attribute {}: {e}", attr.name)))?, - ) + let (osz, lsz, unit) = (handle.offset_size, handle.length_size, conv.vl_unit); + let what = format!("attribute {}", attr.name); + Elements::Vl(handle.with(py, |f| { + resolve_vl(f.storage(), raw, n, osz, lsz, unit).map_err(|e| e.into_py(&what)) + })?) } else { Elements::Bytes(attr.raw_data.clone()) }; diff --git a/crates/clawhdf5-py/src/convert.rs b/crates/clawhdf5-py/src/convert.rs index 4ca7546..c8c3d68 100644 --- a/crates/clawhdf5-py/src/convert.rs +++ b/crates/clawhdf5-py/src/convert.rs @@ -17,6 +17,7 @@ use std::collections::HashMap; use clawhdf5_format::datatype::{CharacterSet, Datatype, DatatypeByteOrder}; use clawhdf5_format::global_heap::GlobalHeapCollection; +use clawhdf5_format::storage::Storage; use numpy::PyArray1; use pyo3::exceptions::{PyTypeError, PyValueError}; use pyo3::prelude::*; @@ -538,16 +539,16 @@ fn object_array<'py>( /// their bytes: each element's stored length times `unit` (1 for strings, /// the base type's size for sequences). Pure Rust, so it runs without the /// GIL. -pub(crate) fn resolve_vl( - file_data: &[u8], +pub(crate) fn resolve_vl( + file: &S, raw: &[u8], count: usize, offset_size: u8, length_size: u8, unit: usize, -) -> Result>, String> { +) -> Result>, VlError> { let refs = clawhdf5_format::vl_data::parse_vl_references(raw, count as u64, offset_size) - .map_err(|e| e.to_string())?; + .map_err(|e| VlError::Invalid(e.to_string()))?; let undefined = match offset_size { 2 => 0xFFFF, 4 => 0xFFFF_FFFF, @@ -558,43 +559,70 @@ pub(crate) fn resolve_vl( for vl in &refs { if vl.collection_address == 0 || vl.collection_address == undefined { if vl.length != 0 { - return Err(format!( + return Err(VlError::Invalid(format!( "variable-length element of length {} has no heap address", vl.length - )); + ))); } out.push(Vec::new()); continue; } let coll = match collections.entry(vl.collection_address) { std::collections::hash_map::Entry::Occupied(e) => e.into_mut(), - std::collections::hash_map::Entry::Vacant(e) => { - let addr = usize::try_from(vl.collection_address) - .map_err(|_| "global heap address out of range".to_string())?; - e.insert( - GlobalHeapCollection::parse(file_data, addr, length_size) - .map_err(|e| e.to_string())?, - ) - } + std::collections::hash_map::Entry::Vacant(e) => e.insert( + GlobalHeapCollection::parse_in(file, vl.collection_address, length_size) + .map_err(VlError::from_format)?, + ), }; - let index = u16::try_from(vl.object_index) - .map_err(|_| format!("global heap object index {} out of range", vl.object_index))?; + let index = u16::try_from(vl.object_index).map_err(|_| { + VlError::Invalid(format!( + "global heap object index {} out of range", + vl.object_index + )) + })?; let obj = coll.get_object(index).ok_or_else(|| { - format!( + VlError::Invalid(format!( "global heap object {index} not found in the collection at {}", vl.collection_address - ) + )) })?; let need = (vl.length as usize) .checked_mul(unit) - .ok_or("variable-length element too long")?; + .ok_or_else(|| VlError::Invalid("variable-length element too long".into()))?; if need > obj.data.len() { - return Err(format!( + return Err(VlError::Invalid(format!( "variable-length element of {need} bytes in a {}-byte heap object", obj.data.len() - )); + ))); } out.push(obj.data[..need].to_vec()); } Ok(out) } + +/// Why variable-length elements could not be resolved. +#[derive(Debug)] +pub(crate) enum VlError { + /// Reading the file failed (a network error on a remote file). + Storage(String), + /// The references or the heap are not valid. + Invalid(String), +} + +impl VlError { + fn from_format(e: clawhdf5_format::error::FormatError) -> Self { + match e { + clawhdf5_format::error::FormatError::Storage(_) => VlError::Storage(e.to_string()), + e => VlError::Invalid(e.to_string()), + } + } + + /// As a Python exception, the message prefixed with `what`: a storage + /// failure is an `OSError`, anything else a `ValueError`. + pub(crate) fn into_py(self, what: &str) -> PyErr { + match self { + VlError::Storage(m) => pyo3::exceptions::PyOSError::new_err(format!("{what}: {m}")), + VlError::Invalid(m) => PyValueError::new_err(format!("{what}: {m}")), + } + } +} diff --git a/crates/clawhdf5-py/src/dataset.rs b/crates/clawhdf5-py/src/dataset.rs index bcfd852..24c9b51 100644 --- a/crates/clawhdf5-py/src/dataset.rs +++ b/crates/clawhdf5-py/src/dataset.rs @@ -6,21 +6,53 @@ //! whole dataset instead); the //! bytes it returns become the numpy array's buffer without a copy (see //! `convert`). All file access and decoding runs with the GIL released, so -//! Python threads reading the same or different datasets run in parallel. +//! Python threads reading the same or different datasets run in parallel, +//! and a remote file's network reads never hold the GIL. use std::sync::Arc; use clawhdf5_format::datatype::Datatype; use clawhdf5_format::object_header::ObjectHeader; +use clawhdf5_rs::File; use pyo3::exceptions::{PyTypeError, PyValueError}; use pyo3::prelude::*; use pyo3::types::{PyList, PyTuple}; use crate::attrs::PyAttrs; -use crate::convert::{Converter, Elements, resolve_vl}; +use crate::convert::{Converter, Elements, VlError, resolve_vl}; +use crate::handle::Handle; use crate::select::{self, Plan}; use crate::{PyEmpty, node, to_py_err}; +/// What opening a dataset reads from the file (without the GIL). +pub(crate) struct DatasetMeta { + /// `None` for a dataset with a null dataspace (h5py's `Empty`). + shape: Option>, + chunks: Option>, + datatype: Datatype, +} + +impl DatasetMeta { + pub(crate) fn load(f: &File, addr: u64, hdr: &ObjectHeader, path: &str) -> PyResult { + let null = node::is_null(&node::dataspace(f, hdr, path)?); + let ds = f.dataset_at(addr).map_err(to_py_err)?; + let shape = if null { + None + } else { + Some(ds.shape().map_err(to_py_err)?) + }; + let datatype = ds.raw_datatype().map_err(to_py_err)?; + let chunks = shape + .as_ref() + .and_then(|s| node::chunk_shape(f, hdr, s.len())); + Ok(Self { + shape, + chunks, + datatype, + }) + } +} + /// A dataset in a file opened for reading. /// /// ```python @@ -30,7 +62,7 @@ use crate::{PyEmpty, node, to_py_err}; /// ``` #[pyclass(name = "Dataset")] pub struct PyDataset { - file: Arc, + handle: Arc, path: String, /// Where the dataset's object header is: reads open it from here rather /// than resolve `path` again. @@ -45,39 +77,24 @@ pub struct PyDataset { } impl PyDataset { - pub(crate) fn open( + pub(crate) fn new( py: Python<'_>, - file: Arc, + handle: Arc, path: String, addr: u64, - hdr: &ObjectHeader, - ) -> PyResult { - crate::no_panic(|| { - let null = node::is_null(&node::dataspace(&file, hdr)?); - let (shape, datatype) = { - let ds = file.dataset_at(addr).map_err(to_py_err)?; - let shape = if null { - None - } else { - Some(ds.shape().map_err(to_py_err)?) - }; - (shape, ds.raw_datatype().map_err(to_py_err)?) - }; - let conv = Converter::new(py, &datatype, file.superblock().offset_size) - .map_err(|e| e.value(py).to_string()); - let chunks = shape - .as_ref() - .and_then(|s| node::chunk_shape(&file, hdr, s.len())); - Ok(Self { - file, - path, - addr, - shape, - chunks, - datatype, - conv, - }) - }) + meta: DatasetMeta, + ) -> Self { + let conv = crate::no_panic(|| Converter::new(py, &meta.datatype, handle.offset_size)) + .map_err(|e| e.value(py).to_string()); + Self { + handle, + path, + addr, + shape: meta.shape, + chunks: meta.chunks, + datatype: meta.datatype, + conv, + } } fn converter(&self) -> PyResult<&Converter> { @@ -103,10 +120,10 @@ impl PyDataset { }; let (reads, list_axis) = plan.reads(dims, chunk_len, elem_size); let read_shape = plan.read_shape(); - let file = &*self.file; + let handle = &*self.handle; let addr = self.addr; // Everything below touches only Rust data: release the GIL. - let read = || -> Result { + let read = |file: &File| -> Result { let ds = file.dataset_at(addr)?; let mut blocks = Vec::with_capacity(reads.len()); for read in reads { @@ -147,7 +164,7 @@ impl PyDataset { let sb = file.superblock(); let n = read_shape.iter().product(); resolve_vl( - file.as_bytes(), + file.storage(), &raw, n, sb.offset_size, @@ -155,12 +172,20 @@ impl PyDataset { unit, ) .map(Elements::Vl) - .map_err(ReadError::Other) + .map_err(ReadError::Vl) }; let data = py .detach(|| { - std::panic::catch_unwind(std::panic::AssertUnwindSafe(read)) - .unwrap_or_else(|p| Err(ReadError::Panic(crate::panic_text(&*p)))) + handle + .with_detached(|f| { + Ok( + std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| read(f))) + .unwrap_or_else(|p| { + Err(ReadError::Panic(crate::panic_text(&*p))) + }), + ) + }) + .unwrap_or_else(|e| Err(ReadError::Py(e))) }) .map_err(|e| e.into_py(&self.path))?; let joined = conv.to_array(py, data, &read_shape, false)?; @@ -183,6 +208,8 @@ impl PyDataset { /// An error from the read closure, turned into a Python error with the GIL. enum ReadError { Lib(clawhdf5_rs::Error), + Vl(VlError), + Py(PyErr), Other(String), Panic(String), } @@ -197,6 +224,8 @@ impl ReadError { fn into_py(self, path: &str) -> PyErr { match self { ReadError::Lib(e) => to_py_err(e), + ReadError::Vl(e) => e.into_py(&node::name(path)), + ReadError::Py(e) => e, ReadError::Other(msg) => PyValueError::new_err(format!("{}: {msg}", node::name(path))), ReadError::Panic(msg) => crate::InternalError::new_err(format!( "{}: clawhdf5 internal error (please report it): {msg}", @@ -252,22 +281,23 @@ impl PyDataset { /// The maximum shape (`None` per unlimited dimension), like h5py. #[getter] fn maxshape<'py>(&self, py: Python<'py>) -> PyResult> { - crate::no_panic(|| { - let Some(shape) = &self.shape else { - return Ok(py.None().into_bound(py)); - }; - let max = self - .file - .dataset_at(self.addr) - .and_then(|ds| ds.max_dimensions()) - .map_err(to_py_err)? - .unwrap_or_else(|| shape.clone()); - let items: Vec> = max - .into_iter() - .map(|d| (d != u64::MAX).then_some(d)) - .collect(); - Ok(PyTuple::new(py, items)?.into_any()) - }) + let Some(shape) = &self.shape else { + return Ok(py.None().into_bound(py)); + }; + let addr = self.addr; + let max = self + .handle + .with(py, |f| { + f.dataset_at(addr) + .and_then(|ds| ds.max_dimensions()) + .map_err(to_py_err) + })? + .unwrap_or_else(|| shape.clone()); + let items: Vec> = max + .into_iter() + .map(|d| (d != u64::MAX).then_some(d)) + .collect(); + Ok(PyTuple::new(py, items)?.into_any()) } /// The dataset's numpy dtype, as h5py reports it. @@ -295,8 +325,8 @@ impl PyDataset { /// The dataset's attributes (read-only, dict-like). #[getter] - fn attrs(&self) -> PyResult { - PyAttrs::read(Arc::clone(&self.file), self.addr, &self.path) + fn attrs(&self, py: Python<'_>) -> PyResult { + PyAttrs::read(py, Arc::clone(&self.handle), self.addr, &self.path) } /// Read with h5py indexing: integers, slices with positive steps, diff --git a/crates/clawhdf5-py/src/file.rs b/crates/clawhdf5-py/src/file.rs index 5f7f51b..48d94c0 100644 --- a/crates/clawhdf5-py/src/file.rs +++ b/crates/clawhdf5-py/src/file.rs @@ -1,13 +1,17 @@ //! PyFile — the main entry point for opening and creating HDF5 files. +use std::collections::HashMap; use std::path::PathBuf; use std::sync::{Arc, Mutex}; +use std::time::Duration; +use pyo3::exceptions::PyValueError; use pyo3::prelude::*; -use pyo3::types::PyList; +use pyo3::types::{PyDict, PyList}; use crate::attrs::PyAttrs; use crate::group::{PyGroup, ReadGroup, WriteGroupState, finalize_write_group}; +use crate::handle::Handle; use crate::{DatasetSpec, OwnedAttrValue, apply_dataset_spec, extract_numpy_data, to_py_err}; /// Internal state for write mode. @@ -23,8 +27,10 @@ struct WriteState { /// Mirrors the h5py.File interface: /// /// ```python -/// # Reading +/// # Reading, a local file or a URL (range requests, nothing downloaded +/// # up front) /// f = clawhdf5.File('data.h5', 'r') +/// f = clawhdf5.File('https://example.org/data.h5') /// ds = f['dataset'] /// f.close() /// @@ -44,27 +50,51 @@ enum FileInner { Write(WriteState), } +/// Whether `s` is a URL (`scheme://…`) rather than a path: the scheme is a +/// letter followed by letters, digits, `+`, `-` or `.` (RFC 3986). +fn is_url(s: &str) -> bool { + let Some((scheme, _)) = s.split_once("://") else { + return false; + }; + let mut chars = scheme.chars(); + chars.next().is_some_and(|c| c.is_ascii_alphabetic()) + && chars.all(|c| c.is_ascii_alphanumeric() || matches!(c, '+' | '-' | '.')) +} + +impl PyFile { + fn from_handle(handle: Arc, filename: String) -> Self { + let root = handle.root; + Self { + inner: Some(FileInner::Read(ReadGroup::new(handle, String::new(), root))), + filename, + } + } +} + #[pymethods] impl PyFile { /// Open or create an HDF5 file. /// /// Parameters: - /// path: file path + /// path: file path, or a URL (`http://`, `https://`, `s3://`, `gs://`, + /// `az://`; which schemes work depends on how the wheel was built) + /// to read the file remotely with default options (see `open_url`) /// mode: 'r' for read (default), 'w' for write #[new] #[pyo3(signature = (path, mode="r"))] fn new(py: Python<'_>, path: &str, mode: &str) -> PyResult { let filename = path.to_string(); - match mode { - "r" => { - let file = py.detach(|| { - crate::no_panic(|| clawhdf5_rs::File::open(path).map_err(to_py_err)) - })?; - Ok(Self { - inner: Some(FileInner::Read(root_group(Arc::new(file)))), - filename, - }) + if is_url(path) { + if mode != "r" { + return Err(PyValueError::new_err(format!( + "remote files are read-only: mode '{mode}' is not supported for a URL" + ))); } + let handle = Handle::open_url(py, path, &clawhdf5_remote::Options::default())?; + return Ok(Self::from_handle(handle, filename)); + } + match mode { + "r" => Ok(Self::from_handle(Handle::open_local(py, path)?, filename)), "w" => Ok(Self { filename, inner: Some(FileInner::Write(WriteState { @@ -74,12 +104,122 @@ impl PyFile { groups: Vec::new(), })), }), - other => Err(PyErr::new::(format!( + other => Err(PyValueError::new_err(format!( "unsupported mode '{other}'; expected 'r' or 'w'" ))), } } + /// Open a remote file for reading, with options. + /// + /// The file is read through a block cache with range requests: opening + /// costs one request (it also fetches the first block), and a read + /// fetches only the blocks it needs. The GIL is released while waiting + /// on the network. + /// + /// Parameters (all optional): + /// block_size: bytes per cached block (default 1 MiB) + /// cache_size: byte budget of the block cache (default 64 MiB) + /// headers: dict of extra HTTP headers (e.g. Authorization), sent only + /// to the URL's own origin + /// retries: retries of a request that failed transiently (default 3) + /// timeout: seconds to connect and receive response headers (default 30) + /// allow_full_download: when the server ignores Range requests, + /// download the whole file once instead of failing (default False) + /// max_full_download: largest file such a download may fetch + /// (default 1 GiB) + /// require_validator: refuse a server that sends neither ETag nor + /// Last-Modified (default False) + /// max_redirects: redirects followed per request (default 5) + /// max_parallel: requests of one read in flight at once (default 8) + #[staticmethod] + #[allow(clippy::too_many_arguments)] + #[pyo3(signature = (url, *, block_size=None, cache_size=None, headers=None, retries=None, + timeout=None, allow_full_download=None, max_full_download=None, + require_validator=None, max_redirects=None, max_parallel=None))] + fn open_url( + py: Python<'_>, + url: &str, + block_size: Option, + cache_size: Option, + headers: Option>, + retries: Option, + timeout: Option, + allow_full_download: Option, + max_full_download: Option, + require_validator: Option, + max_redirects: Option, + max_parallel: Option, + ) -> PyResult { + let mut options = clawhdf5_remote::Options::default(); + if let Some(b) = block_size { + if b == 0 { + return Err(PyValueError::new_err("block_size must be positive")); + } + options.cache.block_size = b; + options.cache.coalesce_gap = b; + // The opening request fetches the first block, not 1 MiB. + options.http.first_request = b; + } + if let Some(c) = cache_size { + options.cache.capacity = c; + } + let http = &mut options.http; + if let Some(h) = headers { + http.headers = h.into_iter().collect(); + } + if let Some(r) = retries { + http.retries = r; + } + if let Some(t) = timeout { + if !(t.is_finite() && t > 0.0) { + return Err(PyValueError::new_err("timeout must be a positive number")); + } + http.timeout = Duration::from_secs_f64(t); + } + if let Some(a) = allow_full_download { + http.allow_full_download = a; + } + if let Some(m) = max_full_download { + http.max_full_download = m; + } + if let Some(v) = require_validator { + http.require_validator = v; + } + if let Some(r) = max_redirects { + http.max_redirects = r; + } + if let Some(p) = max_parallel { + if p == 0 { + return Err(PyValueError::new_err("max_parallel must be positive")); + } + http.max_parallel = p; + } + let handle = Handle::open_url(py, url, &options)?; + Ok(Self::from_handle(handle, url.to_string())) + } + + /// For a remote file, what its block cache has done so far (reads, + /// hits, misses, requests, bytes fetched, ...); `None` for a local file. + #[getter] + fn remote_stats<'py>(&self, py: Python<'py>) -> PyResult>> { + let Some(storage) = self.read_file()?.handle.remote_storage() else { + return Ok(None); + }; + let s = storage.stats(); + let d = PyDict::new(py); + d.set_item("reads", s.reads)?; + d.set_item("hits", s.hits)?; + d.set_item("misses", s.misses)?; + d.set_item("waits", s.waits)?; + d.set_item("requests", s.requests)?; + d.set_item("fetch_calls", s.fetch_calls)?; + d.set_item("bytes_fetched", s.bytes_fetched)?; + d.set_item("evictions", s.evictions)?; + d.set_item("cached_bytes", s.cached_bytes)?; + Ok(Some(d)) + } + /// Close the file. In write mode, this finalizes and writes the file. fn close(&mut self) -> PyResult<()> { let inner = self.inner.take().ok_or_else(|| { @@ -121,7 +261,7 @@ impl PyFile { /// List the names of all children in the root group. fn keys(&self, py: Python<'_>) -> PyResult> { - let names = self.read_file()?.member_names()?; + let names = self.read_file()?.member_names(py)?; Ok(PyList::new(py, names)?.into_any().unbind()) } @@ -139,8 +279,8 @@ impl PyFile { self.keys(py)?.call_method0(py, "__iter__") } - fn __len__(&self) -> PyResult { - Ok(self.read_file()?.member_names()?.len()) + fn __len__(&self, py: Python<'_>) -> PyResult { + Ok(self.read_file()?.member_names(py)?.len()) } /// The root group's name, `/`. @@ -149,7 +289,7 @@ impl PyFile { "/" } - /// The path the file was opened with. + /// The path (or URL) the file was opened with. #[getter] fn filename(&self) -> &str { &self.filename @@ -204,9 +344,9 @@ impl PyFile { /// Attribute access. In read mode, returns attributes of the root group. /// In write mode, returns a writable attrs handle. #[getter] - fn attrs(&self) -> PyResult { + fn attrs(&self, py: Python<'_>) -> PyResult { match self.inner.as_ref() { - Some(FileInner::Read(root)) => root.attrs(), + Some(FileInner::Read(root)) => root.attrs(py), Some(FileInner::Write(state)) => Ok(PyAttrs::from_write(Arc::clone(&state.root_attrs))), None => Err(PyErr::new::( "file is closed", @@ -216,9 +356,10 @@ impl PyFile { fn __repr__(&self) -> String { match &self.inner { - Some(FileInner::Read(root)) => { - format!("", root.file.as_bytes().len()) - } + Some(FileInner::Read(root)) => match root.handle.redacted_url() { + Some(url) => format!(""), + None => format!("", self.filename), + }, Some(FileInner::Write(s)) => { format!("", s.path.display()) } @@ -226,8 +367,8 @@ impl PyFile { } } - fn __contains__(&self, key: &str) -> PyResult { - Ok(self.read_file()?.contains(key)) + fn __contains__(&self, py: Python<'_>, key: &str) -> PyResult { + self.read_file()?.contains(py, key) } } @@ -271,11 +412,6 @@ fn parse_compression( } } -fn root_group(file: Arc) -> ReadGroup { - let root = file.superblock().root_group_address; - ReadGroup::new(file, String::new(), root) -} - /// Build and write the HDF5 file from accumulated write state. fn finalize_write(state: WriteState) -> PyResult<()> { crate::no_panic(|| { @@ -309,6 +445,18 @@ fn finalize_write(state: WriteState) -> PyResult<()> { mod tests { use super::*; + #[test] + fn urls_and_paths() { + assert!(is_url("http://h/f.h5")); + assert!(is_url("s3://bucket/key.h5")); + assert!(is_url("git+https://x")); + assert!(!is_url("data.h5")); + assert!(!is_url("/tmp/a://b.h5")); + assert!(!is_url("dir/x://y")); + assert!(!is_url("1http://x")); + assert!(!is_url("://x")); + } + #[test] fn parse_gzip_compression() { assert_eq!(parse_compression(Some("gzip"), Some(6)).unwrap(), Some(6)); diff --git a/crates/clawhdf5-py/src/group.rs b/crates/clawhdf5-py/src/group.rs index 7585e57..36506b9 100644 --- a/crates/clawhdf5-py/src/group.rs +++ b/crates/clawhdf5-py/src/group.rs @@ -3,11 +3,12 @@ use std::collections::HashMap; use std::sync::{Arc, Mutex, OnceLock}; -use pyo3::exceptions::{PyIOError, PyKeyError, PyValueError}; +use pyo3::exceptions::{PyIOError, PyKeyError, PyOSError, PyValueError}; use pyo3::prelude::*; use pyo3::types::PyList; use crate::attrs::PyAttrs; +use crate::handle::Handle; use crate::{DatasetSpec, OwnedAttrValue, apply_dataset_spec, extract_numpy_data, node}; /// Shared state for a group being written. @@ -34,9 +35,9 @@ enum GroupInner { } impl PyGroup { - pub(crate) fn from_read(file: Arc, path: String, addr: u64) -> Self { + pub(crate) fn from_read(handle: Arc, path: String, addr: u64) -> Self { Self { - inner: GroupInner::Read(ReadGroup::new(file, path, addr)), + inner: GroupInner::Read(ReadGroup::new(handle, path, addr)), } } @@ -60,9 +61,10 @@ impl PyGroup { /// h5py). It keeps its own address and, once listed, its links, so looking /// up a child neither resolves the path from the root nor scans the group's /// links again: visiting every member of a large group is linear, not -/// quadratic. +/// quadratic. (Edits never add or remove links, so these stay valid in a +/// file open for editing.) pub(crate) struct ReadGroup { - pub file: Arc, + pub handle: Arc, pub path: String, pub addr: u64, /// Link name -> object address (soft links resolved), filled on first use. @@ -72,9 +74,9 @@ pub(crate) struct ReadGroup { } impl ReadGroup { - pub(crate) fn new(file: Arc, path: String, addr: u64) -> Self { + pub(crate) fn new(handle: Arc, path: String, addr: u64) -> Self { Self { - file, + handle, path, addr, links: OnceLock::new(), @@ -82,17 +84,14 @@ impl ReadGroup { } } - fn links(&self) -> PyResult<&HashMap> { + fn links(&self, py: Python<'_>) -> PyResult<&HashMap> { if let Some(links) = self.links.get() { return Ok(links); } - let entries = crate::no_panic(|| { - clawhdf5_format::group_v2::resolve_group_children( - self.file.as_bytes(), - self.file.superblock(), - self.addr, - ) - .map_err(|e| PyValueError::new_err(format!("{}: {e}", node::name(&self.path)))) + let (addr, path) = (self.addr, &self.path); + let entries = self.handle.with(py, |f| { + clawhdf5_format::group_v2::resolve_group_children_in(f.storage(), f.superblock(), addr) + .map_err(|e| node::format_err(path, e, PyValueError::new_err)) })?; let map = entries .into_iter() @@ -102,7 +101,7 @@ impl ReadGroup { } /// The path and address of `key` (a name, a relative or an absolute path). - fn locate(&self, key: &str) -> PyResult<(String, u64)> { + fn locate(&self, py: Python<'_>, key: &str) -> PyResult<(String, u64)> { let path = node::join(&self.path, key); let rel = if self.path.is_empty() { Some(path.as_str()) @@ -112,24 +111,24 @@ impl ReadGroup { path.strip_prefix(self.path.as_str()) .and_then(|r| r.strip_prefix('/')) }; - let addr = match rel { - // A direct child: the link table, when it has the name. - Some(name) if !name.is_empty() && !name.contains('/') => { - match self.links()?.get(name) { - Some(&a) => a, - None => node::resolve_from(&self.file, self.addr, name, &path)?, - } - } - Some(rel) => node::resolve_from(&self.file, self.addr, rel, &path)?, - None => node::address(&self.file, &path)?, - }; - Ok((path, addr)) + // A direct child: the link table, when it has the name. + if let Some(name) = rel.filter(|n| !n.is_empty() && !n.contains('/')) + && let Some(&a) = self.links(py)?.get(name) + { + return Ok((path, a)); + } + let addr = self.addr; + let found = self.handle.with(py, |f| match rel { + Some(rel) => node::resolve_from(f, addr, rel, &path), + None => node::address(f, &path), + })?; + Ok((path, found)) } /// `group[key]`. pub(crate) fn get_item(&self, py: Python<'_>, key: &str) -> PyResult> { - let (path, addr) = self.locate(key)?; - node::open(py, &self.file, path, addr) + let (path, addr) = self.locate(py, key)?; + node::open(py, &self.handle, path, addr) } /// `group.get(key, default)`. @@ -148,48 +147,57 @@ impl ReadGroup { } /// Names of the group's datasets and subgroups, sorted (h5py's order). - pub(crate) fn member_names(&self) -> PyResult<&[String]> { + pub(crate) fn member_names(&self, py: Python<'_>) -> PyResult<&[String]> { if let Some(m) = self.members.get() { return Ok(m); } - let mut names = Vec::new(); - for (name, &addr) in self.links()? { - let hdr = node::header_at(&self.file, addr, &node::join(&self.path, name))?; - if matches!( - node::kind(&hdr), - Some(node::Kind::Dataset | node::Kind::Group) - ) { - names.push(name.clone()); + let links = self.links(py)?; + let path = &self.path; + let mut names = self.handle.with(py, |f| { + let mut names = Vec::new(); + for (name, &addr) in links { + if matches!( + node::kind_at(f, addr, &node::join(path, name))?, + Some(node::Kind::Dataset | node::Kind::Group) + ) { + names.push(name.clone()); + } } - } + Ok(names) + })?; names.sort_by(|a, b| a.as_bytes().cmp(b.as_bytes())); Ok(self.members.get_or_init(|| names)) } - pub(crate) fn contains(&self, key: &str) -> bool { - self.locate(key) - .and_then(|(path, addr)| node::header_at(&self.file, addr, &path)) - .ok() - .and_then(|h| node::kind(&h)) - .is_some_and(|k| k != node::Kind::Datatype) + /// `key in group`: whether `key` names a dataset or group. A failed + /// read of the file (a network error) is raised, not `False`. + pub(crate) fn contains(&self, py: Python<'_>, key: &str) -> PyResult { + let found = self + .locate(py, key) + .and_then(|(path, addr)| self.handle.with(py, |f| node::kind_at(f, addr, &path))); + match found { + Ok(kind) => Ok(kind.is_some_and(|k| k != node::Kind::Datatype)), + Err(e) if e.is_instance_of::(py) => Err(e), + Err(_) => Ok(false), + } } pub(crate) fn values(&self, py: Python<'_>) -> PyResult>> { - self.member_names()? + self.member_names(py)? .iter() .map(|n| self.get_item(py, n)) .collect() } pub(crate) fn items(&self, py: Python<'_>) -> PyResult)>> { - self.member_names()? + self.member_names(py)? .iter() .map(|n| Ok((n.clone(), self.get_item(py, n)?))) .collect() } - pub(crate) fn attrs(&self) -> PyResult { - PyAttrs::read(Arc::clone(&self.file), self.addr, &self.path) + pub(crate) fn attrs(&self, py: Python<'_>) -> PyResult { + PyAttrs::read(py, Arc::clone(&self.handle), self.addr, &self.path) } } @@ -210,7 +218,7 @@ impl PyGroup { fn keys(&self, py: Python<'_>) -> PyResult> { match &self.inner { GroupInner::Read(g) => { - let list = PyList::new(py, g.member_names()?)?; + let list = PyList::new(py, g.member_names(py)?)?; Ok(list.into_any().unbind()) } GroupInner::Write(state) => { @@ -236,9 +244,9 @@ impl PyGroup { self.keys(py)?.call_method0(py, "__iter__") } - fn __len__(&self) -> PyResult { + fn __len__(&self, py: Python<'_>) -> PyResult { match &self.inner { - GroupInner::Read(g) => Ok(g.member_names()?.len()), + GroupInner::Read(g) => Ok(g.member_names(py)?.len()), GroupInner::Write(state) => Ok(state.lock().unwrap().datasets.len()), } } @@ -301,9 +309,9 @@ impl PyGroup { /// Attribute access. #[getter] - fn attrs(&self) -> PyResult { + fn attrs(&self, py: Python<'_>) -> PyResult { match &self.inner { - GroupInner::Read(g) => g.attrs(), + GroupInner::Read(g) => g.attrs(py), GroupInner::Write(state) => { let store = Arc::clone(&state.lock().unwrap().attrs); Ok(PyAttrs::from_write(store)) @@ -311,10 +319,10 @@ impl PyGroup { } } - fn __repr__(&self) -> String { + fn __repr__(&self, py: Python<'_>) -> String { match &self.inner { GroupInner::Read(g) => { - let n = g.member_names().map_or(0, |m| m.len()); + let n = g.member_names(py).map_or(0, |m| m.len()); format!("", node::name(&g.path)) } GroupInner::Write(state) => { @@ -324,9 +332,9 @@ impl PyGroup { } } - fn __contains__(&self, key: &str) -> PyResult { + fn __contains__(&self, py: Python<'_>, key: &str) -> PyResult { match &self.inner { - GroupInner::Read(g) => Ok(g.contains(key)), + GroupInner::Read(g) => g.contains(py, key), GroupInner::Write(state) => { let guard = state.lock().unwrap(); Ok(guard.datasets.iter().any(|d| d.name == key)) @@ -357,31 +365,6 @@ pub(crate) fn finalize_write_group( mod tests { use super::*; - #[test] - fn member_names_are_sorted() { - let mut b = clawhdf5_rs::FileBuilder::new(); - b.create_dataset("zeta").with_f64_data(&[1.0]); - b.create_dataset("alpha").with_f64_data(&[1.0]); - let mut g = b.create_group("mid"); - g.create_dataset("x").with_f64_data(&[1.0]); - let finished = g.finish(); - b.add_group(finished); - let bytes = b.finish().unwrap(); - let file = Arc::new(clawhdf5_rs::File::from_bytes(bytes).unwrap()); - let root = file.superblock().root_group_address; - let top = ReadGroup::new(Arc::clone(&file), String::new(), root); - assert_eq!(top.member_names().unwrap(), ["alpha", "mid", "zeta"]); - let (path, addr) = top.locate("mid").unwrap(); - assert_eq!(path, "mid"); - let mid = ReadGroup::new(Arc::clone(&file), path, addr); - assert_eq!(mid.member_names().unwrap(), ["x"]); - assert!(top.contains("mid/x")); - assert!(mid.contains("/alpha")); - assert!(mid.contains("x") && mid.contains("./x")); - assert!(!top.contains("nope")); - assert!(!mid.contains("alpha")); - } - #[test] fn finalize_group() { let state = WriteGroupState { diff --git a/crates/clawhdf5-py/src/handle.rs b/crates/clawhdf5-py/src/handle.rs new file mode 100644 index 0000000..8649f74 --- /dev/null +++ b/crates/clawhdf5-py/src/handle.rs @@ -0,0 +1,130 @@ +//! The open file every object of a `File` shares. +//! +//! Every read goes through [`Handle::with`], which releases the GIL and +//! parses through `File::storage()`, so the same code serves a local file +//! (memory-mapped) and a remote one (`clawhdf5-remote`: range requests +//! through a block cache, so a network read never holds the GIL). +//! +//! Lock discipline (no deadlock with the GIL): the file lock is only taken +//! with the GIL released, and code that holds it never touches Python. + +use std::path::PathBuf; +use std::sync::{Arc, PoisonError, RwLock}; + +use clawhdf5_rs::File; +use pyo3::exceptions::PyOSError; +use pyo3::prelude::*; + +use crate::to_py_err; + +/// Where the file's bytes come from. +pub(crate) enum Source { + /// A local path (memory-mapped). + Local(#[allow(dead_code)] PathBuf), + /// A URL, read through `clawhdf5-remote`'s block cache. + Remote { + url: String, + storage: Arc, + }, +} + +pub(crate) struct Handle { + file: RwLock>, + source: Source, + pub offset_size: u8, + pub length_size: u8, + pub root: u64, +} + +fn closed_after_failed_reopen() -> PyErr { + PyOSError::new_err("the file could not be reopened after an edit; open it again") +} + +impl Handle { + fn new(file: File, source: Source) -> Arc { + let sb = file.superblock(); + let (offset_size, length_size, root) = + (sb.offset_size, sb.length_size, sb.root_group_address); + Arc::new(Self { + file: RwLock::new(Some(file)), + source, + offset_size, + length_size, + root, + }) + } + + /// A local file, read-only. + pub(crate) fn open_local(py: Python<'_>, path: &str) -> PyResult> { + let file = py.detach(|| crate::no_panic(|| File::open(path).map_err(to_py_err)))?; + Ok(Self::new(file, Source::Local(PathBuf::from(path)))) + } + + /// A remote file (`http(s)://`, `s3://`, ...). + pub(crate) fn open_url( + py: Python<'_>, + url: &str, + options: &clawhdf5_remote::Options, + ) -> PyResult> { + let (file, storage) = py.detach(|| { + crate::no_panic(|| { + let storage = clawhdf5_remote::storage_for_url(url, options).map_err(remote_err)?; + let file = File::open_storage(storage.clone()).map_err(to_py_err)?; + Ok((file, storage)) + }) + })?; + Ok(Self::new( + file, + Source::Remote { + url: url.to_string(), + storage, + }, + )) + } + + /// Run `f` on the file with the GIL released (a remote read may wait + /// on the network; other Python threads run meanwhile). `f` must not + /// touch Python. + pub(crate) fn with( + &self, + py: Python<'_>, + f: impl FnOnce(&File) -> PyResult + Send, + ) -> PyResult { + py.detach(|| self.with_detached(f)) + } + + /// [`with`](Self::with) for code that already runs without the GIL. + pub(crate) fn with_detached(&self, f: impl FnOnce(&File) -> PyResult) -> PyResult { + crate::no_panic(|| { + let guard = self.file.read().unwrap_or_else(PoisonError::into_inner); + let file = guard.as_ref().ok_or_else(closed_after_failed_reopen)?; + f(file) + }) + } + + /// The remote file's block cache. + pub(crate) fn remote_storage(&self) -> Option<&clawhdf5_remote::RemoteStorage> { + match &self.source { + Source::Remote { storage, .. } => Some(storage), + Source::Local(_) => None, + } + } + + /// The URL of a remote file, credentials and query values redacted. + pub(crate) fn redacted_url(&self) -> Option { + match &self.source { + Source::Remote { url, .. } => Some(clawhdf5_remote::redact_url(url)), + Source::Local(_) => None, + } + } +} + +/// A `clawhdf5_remote::Error` as a Python exception: the network side +/// (unreachable, a status, no range support, a changed file) is `OSError`, +/// a file that is not HDF5 is what `to_py_err` makes of it. +pub(crate) fn remote_err(e: clawhdf5_remote::Error) -> PyErr { + match e { + clawhdf5_remote::Error::Hdf5(e) => to_py_err(e), + other => PyOSError::new_err(other.to_string()), + } +} diff --git a/crates/clawhdf5-py/src/lib.rs b/crates/clawhdf5-py/src/lib.rs index 4055b82..db89d8d 100644 --- a/crates/clawhdf5-py/src/lib.rs +++ b/crates/clawhdf5-py/src/lib.rs @@ -14,6 +14,7 @@ mod convert; mod dataset; mod file; mod group; +mod handle; mod node; mod select; @@ -63,7 +64,7 @@ fn _panic_for_test() -> PyResult<()> { /// Convert a `clawhdf5_rs::Error` into a `PyErr`. /// /// Maps different error variants to more specific Python exception types: -/// - I/O errors -> `PyIOError` +/// - I/O errors, and failed reads of a remote file -> `PyIOError`/`PyOSError` /// - Format/parsing errors -> `PyValueError` /// - Missing dataset/path errors -> `PyKeyError` /// - Invalid arguments -> `PyValueError` @@ -73,6 +74,10 @@ pub(crate) fn to_py_err(e: clawhdf5_rs::Error) -> PyErr { use clawhdf5_rs::Error; match &e { Error::Io(_) => PyErr::new::(e.to_string()), + // A failed read of the storage: a network error on a remote file. + Error::Format(clawhdf5_format::error::FormatError::Storage(_)) => { + PyErr::new::(e.to_string()) + } Error::Format(_) => PyErr::new::(e.to_string()), Error::NotADataset(_) | Error::MissingMessage(_) => { PyErr::new::(e.to_string()) diff --git a/crates/clawhdf5-py/src/node.rs b/crates/clawhdf5-py/src/node.rs index fa9ede7..35e5dec 100644 --- a/crates/clawhdf5-py/src/node.rs +++ b/crates/clawhdf5-py/src/node.rs @@ -1,16 +1,24 @@ //! Resolving paths to objects in a file opened for reading. +//! +//! Everything here parses through `File::storage()` (the `clawhdf5_format` +//! `*_in` functions), never `File::as_bytes()`, so it works the same on a +//! memory-mapped local file and on a remote one; and it runs inside +//! `Handle::with`, without the GIL. use std::sync::Arc; use clawhdf5_format::attribute::AttributeMessage; use clawhdf5_format::dataspace::{Dataspace, DataspaceType}; +use clawhdf5_format::error::FormatError; use clawhdf5_format::message_type::MessageType; use clawhdf5_format::object_header::ObjectHeader; -use pyo3::exceptions::{PyKeyError, PyTypeError, PyValueError}; +use clawhdf5_rs::File; +use pyo3::exceptions::{PyKeyError, PyOSError, PyTypeError, PyValueError}; use pyo3::prelude::*; -use crate::dataset::PyDataset; +use crate::dataset::{DatasetMeta, PyDataset}; use crate::group::PyGroup; +use crate::handle::Handle; /// Join `key` onto the group path `base` the way h5py does: an absolute key /// starts from the root, a relative one from `base`. Paths are kept without @@ -33,42 +41,46 @@ pub(crate) fn name(path: &str) -> String { format!("/{path}") } +/// A format error met at `path`: a failed read of the storage (a network +/// error on a remote file) is an `OSError`, anything else `other(message)`. +pub(crate) fn format_err(path: &str, e: FormatError, other: fn(String) -> PyErr) -> PyErr { + let msg = format!("{}: {e}", name(path)); + match e { + FormatError::Storage(_) => PyOSError::new_err(msg), + _ => other(msg), + } +} + +fn value_err(msg: String) -> PyErr { + PyValueError::new_err(msg) +} + /// The address of the object at `path`, resolved from the root group. -pub(crate) fn address(file: &clawhdf5_rs::File, path: &str) -> PyResult { +pub(crate) fn address(file: &File, path: &str) -> PyResult { resolve_from(file, file.superblock().root_group_address, path, path) } /// The address of `rel` resolved from the group at `group` (`full` is the /// resulting path, for the error message). -pub(crate) fn resolve_from( - file: &clawhdf5_rs::File, - group: u64, - rel: &str, - full: &str, -) -> PyResult { +pub(crate) fn resolve_from(file: &File, group: u64, rel: &str, full: &str) -> PyResult { if rel.is_empty() { return Ok(group); } - crate::no_panic(|| { - clawhdf5_format::group_v2::resolve_path_from(file.as_bytes(), file.superblock(), group, rel) - .map_err(|e| { - PyKeyError::new_err(format!( - "Unable to open object (object '{}' doesn't exist): {e}", - name(full) - )) - }) - }) + clawhdf5_format::group_v2::resolve_path_from_in(file.storage(), file.superblock(), group, rel) + .map_err(|e| match e { + FormatError::Storage(_) => format_err(full, e, value_err), + e => PyKeyError::new_err(format!( + "Unable to open object (object '{}' doesn't exist): {e}", + name(full) + )), + }) } /// The object header at `addr` (the object at `path`). -pub(crate) fn header_at(file: &clawhdf5_rs::File, addr: u64, path: &str) -> PyResult { - crate::no_panic(|| { - let sb = file.superblock(); - let at = usize::try_from(addr) - .map_err(|_| PyValueError::new_err(format!("{}: address out of range", name(path))))?; - ObjectHeader::parse(file.as_bytes(), at, sb.offset_size, sb.length_size) - .map_err(|e| PyValueError::new_err(format!("{}: {e}", name(path)))) - }) +pub(crate) fn header_at(file: &File, addr: u64, path: &str) -> PyResult { + let sb = file.superblock(); + ObjectHeader::parse_in(file.storage(), addr, sb.offset_size, sb.length_size) + .map_err(|e| format_err(path, e, value_err)) } /// What an object header describes. @@ -96,29 +108,50 @@ pub(crate) fn kind(hdr: &ObjectHeader) -> Option { } } +/// The kind of the object at `addr`, from its header. +pub(crate) fn kind_at(file: &File, addr: u64, path: &str) -> PyResult> { + Ok(kind(&header_at(file, addr, path)?)) +} + +/// What opening an object found, read without the GIL. +enum Found { + Dataset(DatasetMeta), + Group, + Datatype, + Other, +} + /// Open the object at `addr` (whose path is `path`) as a `Dataset` or /// `Group`. Both keep the address, so later reads resolve nothing. pub(crate) fn open( py: Python<'_>, - file: &Arc, + handle: &Arc, path: String, addr: u64, ) -> PyResult> { - let hdr = header_at(file, addr, &path)?; - match kind(&hdr) { - Some(Kind::Dataset) => Ok(PyDataset::open(py, Arc::clone(file), path, addr, &hdr)? + let found = handle.with(py, |f| { + let hdr = header_at(f, addr, &path)?; + Ok(match kind(&hdr) { + Some(Kind::Dataset) => Found::Dataset(DatasetMeta::load(f, addr, &hdr, &path)?), + Some(Kind::Group) => Found::Group, + Some(Kind::Datatype) => Found::Datatype, + None => Found::Other, + }) + })?; + match found { + Found::Dataset(meta) => Ok(PyDataset::new(py, Arc::clone(handle), path, addr, meta) .into_pyobject(py)? .into_any() .unbind()), - Some(Kind::Group) => Ok(PyGroup::from_read(Arc::clone(file), path, addr) + Found::Group => Ok(PyGroup::from_read(Arc::clone(handle), path, addr) .into_pyobject(py)? .into_any() .unbind()), - Some(Kind::Datatype) => Err(PyTypeError::new_err(format!( + Found::Datatype => Err(PyTypeError::new_err(format!( "{}: committed (named) datatypes are not supported by clawhdf5", name(&path) ))), - None => Err(PyValueError::new_err(format!( + Found::Other => Err(PyValueError::new_err(format!( "{}: not a dataset, group or datatype", name(&path) ))), @@ -126,32 +159,26 @@ pub(crate) fn open( } /// The dataspace message of an object header. -pub(crate) fn dataspace(file: &clawhdf5_rs::File, hdr: &ObjectHeader) -> PyResult { - crate::no_panic(|| { - let sb = file.superblock(); - let msg = hdr - .messages - .iter() - .find(|m| m.msg_type == MessageType::Dataspace) - .ok_or_else(|| PyValueError::new_err("object has no dataspace message"))?; - let data = clawhdf5_format::shared_message::message_data( - file.as_bytes(), - msg, - sb.offset_size, - sb.length_size, - ) - .map_err(|e| PyValueError::new_err(e.to_string()))?; - Dataspace::parse(&data, sb.length_size).map_err(|e| PyValueError::new_err(e.to_string())) - }) +pub(crate) fn dataspace(file: &File, hdr: &ObjectHeader, path: &str) -> PyResult { + let sb = file.superblock(); + let msg = hdr + .messages + .iter() + .find(|m| m.msg_type == MessageType::Dataspace) + .ok_or_else(|| PyValueError::new_err("object has no dataspace message"))?; + let data = clawhdf5_format::shared_message::message_data_in( + file.storage(), + msg, + sb.offset_size, + sb.length_size, + ) + .map_err(|e| format_err(path, e, value_err))?; + Dataspace::parse(&data, sb.length_size).map_err(|e| format_err(path, e, value_err)) } /// The chunk shape of a chunked dataset (one entry per dataset dimension), /// or `None` for other layouts or a layout message that does not parse. -pub(crate) fn chunk_shape( - file: &clawhdf5_rs::File, - hdr: &ObjectHeader, - rank: usize, -) -> Option> { +pub(crate) fn chunk_shape(file: &File, hdr: &ObjectHeader, rank: usize) -> Option> { let sb = file.superblock(); let msg = hdr .messages @@ -179,24 +206,18 @@ pub(crate) fn is_null(space: &Dataspace) -> bool { /// The attributes of the object at `addr` (whose path is `path`), sorted by /// name (h5py's order). Attributes whose messages cannot be parsed are left /// out, as the facade's `attrs()` does. -pub(crate) fn attributes( - file: &clawhdf5_rs::File, - addr: u64, - path: &str, -) -> PyResult> { +pub(crate) fn attributes(file: &File, addr: u64, path: &str) -> PyResult> { let hdr = header_at(file, addr, path)?; - crate::no_panic(|| { - let sb = file.superblock(); - let (mut attrs, _errors) = clawhdf5_format::attribute::extract_attributes_tolerant( - file.as_bytes(), - &hdr, - sb.offset_size, - sb.length_size, - ) - .map_err(|e| PyValueError::new_err(format!("{}: {e}", name(path))))?; - attrs.sort_by(|a, b| a.name.as_bytes().cmp(b.name.as_bytes())); - Ok(attrs) - }) + let sb = file.superblock(); + let (mut attrs, _errors) = clawhdf5_format::attribute::extract_attributes_tolerant_in( + file.storage(), + &hdr, + sb.offset_size, + sb.length_size, + ) + .map_err(|e| format_err(path, e, value_err))?; + attrs.sort_by(|a, b| a.name.as_bytes().cmp(b.name.as_bytes())); + Ok(attrs) } #[cfg(test)] diff --git a/crates/clawhdf5-py/tests/conftest.py b/crates/clawhdf5-py/tests/conftest.py index 967f1e0..88113e8 100644 --- a/crates/clawhdf5-py/tests/conftest.py +++ b/crates/clawhdf5-py/tests/conftest.py @@ -1,6 +1,10 @@ """Shared fixtures for the clawhdf5 Python binding tests.""" import os +import re +import threading +import time +from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer import pytest @@ -16,3 +20,134 @@ def h5py(): pytest.fail("h5py is required (CLAWHDF5_REQUIRE_INTEROP=1) but not importable") pytest.skip("h5py not installed") return mod + + +# --------------------------------------------------------------------------- +# An HTTP server for remote reads +# --------------------------------------------------------------------------- + +_RANGE = re.compile(r"^bytes=(\d*)-(\d*)$") + + +class RangeServer: + """A static file server on 127.0.0.1, in a thread of this process, that + answers `Range: bytes=a-b` with 206 and `Content-Range` (the way S3 and + common web servers do), sends an ETag and honours `If-Match`. + + - `ranges=False`: ignores `Range` and answers 200 with the whole file, + like a server without range support. + - `down` (set by `close()`): hang up on every request. + - `delay`: seconds to wait before answering each request after the + first `delay_after` ones (a slow network). + - `log`: every request as `(method, path, range header)`. + """ + + def __init__(self, root, ranges=True): + self.root = str(root) + self.ranges = ranges + self.delay = 0.0 + self.delay_after = 0 + self.down = False + self.log = [] + self._lock = threading.Lock() + server = self + + class Handler(BaseHTTPRequestHandler): + protocol_version = "HTTP/1.1" + + def log_message(self, *args): # quiet + pass + + def do_HEAD(self): + self._serve(body=False) + + def do_GET(self): + self._serve(body=True) + + def _serve(self, body): + with server._lock: + server.log.append((self.command, self.path, self.headers.get("Range"))) + n = len(server.log) + if server.delay and n > server.delay_after: + time.sleep(server.delay) + if server.down: + # Hang up without an answer (open keep-alive + # connections outlive shutdown(), so close() sets this). + self.close_connection = True + return + path = os.path.join(server.root, self.path.lstrip("/").split("?")[0]) + if not os.path.isfile(path): + self.send_response(404) + self.send_header("Content-Length", "0") + self.end_headers() + return + with open(path, "rb") as fh: + data = fh.read() + st = os.stat(path) + etag = f'"{st.st_mtime_ns:x}-{st.st_size:x}"' + want = self.headers.get("If-Match") + if want is not None and want != etag and want != "*": + self.send_response(412) + self.send_header("Content-Length", "0") + self.end_headers() + return + rng = self.headers.get("Range") if server.ranges else None + m = _RANGE.match(rng.strip()) if rng else None + if m and (m.group(1) or m.group(2)): + size = len(data) + if m.group(1): + start = int(m.group(1)) + end = int(m.group(2)) if m.group(2) else size - 1 + else: + start = max(0, size - int(m.group(2))) + end = size - 1 + if start >= size: + self.send_response(416) + self.send_header("Content-Range", f"bytes */{size}") + self.send_header("Content-Length", "0") + self.end_headers() + return + end = min(end, size - 1) + part = data[start : end + 1] + self.send_response(206) + self.send_header("Content-Range", f"bytes {start}-{end}/{size}") + else: + part = data + self.send_response(200) + if server.ranges: + self.send_header("Accept-Ranges", "bytes") + self.send_header("ETag", etag) + self.send_header("Content-Length", str(len(part))) + self.send_header("Content-Type", "application/x-hdf5") + self.end_headers() + if body: + try: + self.wfile.write(part) + except (BrokenPipeError, ConnectionResetError): + pass + + self.httpd = ThreadingHTTPServer(("127.0.0.1", 0), Handler) + self.httpd.daemon_threads = True + self.port = self.httpd.server_address[1] + self.thread = threading.Thread(target=self.httpd.serve_forever, daemon=True) + self.thread.start() + + def url(self, name): + return f"http://127.0.0.1:{self.port}/{name}" + + def requests(self): + with self._lock: + return len(self.log) + + def close(self): + self.down = True + self.httpd.shutdown() + self.httpd.server_close() + + +@pytest.fixture +def range_server(tmp_path): + """A range-capable server over `tmp_path`.""" + server = RangeServer(tmp_path) + yield server + server.close() diff --git a/crates/clawhdf5-py/tests/test_read_vs_h5py.py b/crates/clawhdf5-py/tests/test_read_vs_h5py.py index 6a3b6cb..f3132a7 100644 --- a/crates/clawhdf5-py/tests/test_read_vs_h5py.py +++ b/crates/clawhdf5-py/tests/test_read_vs_h5py.py @@ -178,15 +178,32 @@ def _write_fixture(h5py, path): g.attrs["depth"] = np.int8(3) -@pytest.fixture(scope="module") -def pair(h5py, tmp_path_factory): - path = str(tmp_path_factory.mktemp("h5") / "fixture.h5") +@pytest.fixture(scope="module", params=["local", "http", "http-1k-blocks"]) +def pair(request, h5py, tmp_path_factory): + """The fixture file through h5py and through clawhdf5: opened locally, + and over HTTP range requests (a local server in this process) with the + default 1 MiB blocks and with 1 KiB blocks, so every structure is read + through many small ranges.""" + from conftest import RangeServer + + root = tmp_path_factory.mktemp("h5") + path = str(root / "fixture.h5") _write_fixture(h5py, path) theirs = h5py.File(path, "r") - ours = clawhdf5.File(path, "r") + server = None + if request.param == "local": + ours = clawhdf5.File(path, "r") + else: + server = RangeServer(root) + if request.param == "http": + ours = clawhdf5.File(server.url("fixture.h5")) + else: + ours = clawhdf5.File.open_url(server.url("fixture.h5"), block_size=1024) yield ours, theirs, path theirs.close() ours.close() + if server is not None: + server.close() def _all_datasets(h5py, f): diff --git a/crates/clawhdf5-py/tests/test_remote.py b/crates/clawhdf5-py/tests/test_remote.py new file mode 100644 index 0000000..dbb24b8 --- /dev/null +++ b/crates/clawhdf5-py/tests/test_remote.py @@ -0,0 +1,244 @@ +"""Remote files: `clawhdf5.File(url)` / `File.open_url(url, ...)` read over +HTTP range requests (clawhdf5-remote's block cache), against a server in this +process (conftest.RangeServer). Values are compared with h5py reading the +same file locally; the rest checks what the server saw (only the blocks a +read needs are fetched), the failure modes (no range support, a missing +file, a file that changes, a server that goes away: errors, never wrong +data), and that the GIL is released while a read waits on the network.""" + +import os +import sys +import threading +import time + +import numpy as np +import pytest + +import clawhdf5 +from conftest import RangeServer + + +def _write(h5py, path): + rng = np.random.default_rng(7) + with h5py.File(path, "w") as f: + f.create_dataset("contig", data=rng.standard_normal((400, 300))) + f.create_dataset( + "chunked", + data=rng.integers(0, 1000, size=(512, 512), dtype="= 2 + + +def test_a_small_read_fetches_only_its_blocks(h5py, remote_file): + """With 4 KiB blocks, opening and reading one chunk of a 1 MB chunked + dataset costs a handful of requests and a few blocks, not the file.""" + path, url, server = remote_file + size = os.path.getsize(path) + f = clawhdf5.File.open_url(url, block_size=4096) + opened = server.requests() + assert opened == 1, server.log + ds = f["chunked"] + got = ds[0:10, 0:10] + with h5py.File(path, "r") as theirs: + np.testing.assert_array_equal(got, theirs["chunked"][0:10, 0:10]) + stats = f.remote_stats + assert stats["bytes_fetched"] < size / 4, (stats, size) + assert server.requests() - opened <= 12, server.log + # A second read of the same region is served by the cache. + before = server.requests() + ds[0:10, 0:10] + assert server.requests() == before + assert f.remote_stats["hits"] > stats["hits"] + assert clawhdf5.File(str(path), "r").remote_stats is None + + +def test_server_without_range_support(h5py, tmp_path): + """A server that ignores Range answers 200 with the whole file: that is + an OSError by default, and a whole download when allowed.""" + path = tmp_path / "remote.h5" + _write(h5py, str(path)) + server = RangeServer(tmp_path, ranges=False) + try: + url = server.url("remote.h5") + with pytest.raises(OSError, match="range"): + clawhdf5.File(url) + with clawhdf5.File.open_url(url, allow_full_download=True) as ours, h5py.File(path, "r") as theirs: + np.testing.assert_array_equal(ours["chunked"][...], theirs["chunked"][...]) + np.testing.assert_array_equal(ours["contig"][5], theirs["contig"][5]) + with pytest.raises(OSError): + clawhdf5.File.open_url(url, allow_full_download=True, max_full_download=1000) + finally: + server.close() + + +def test_errors_are_oserrors(remote_file): + _, url, server = remote_file + with pytest.raises(OSError, match="404"): + clawhdf5.File(server.url("missing.h5")) + with pytest.raises(ValueError, match="read-only"): + clawhdf5.File(url, "r+") + with pytest.raises(ValueError, match="read-only"): + clawhdf5.File(url, "w") + with pytest.raises(OSError, match="unsupported URL"): + clawhdf5.File("nosuchscheme://x/y.h5") + with pytest.raises(ValueError): + clawhdf5.File.open_url(url, block_size=0) + with pytest.raises(TypeError): + clawhdf5.File.open_url(url, no_such_option=1) + + +def test_object_store_urls_need_their_features(): + """The default wheel has no S3/GCS/Azure clients (aws-lc-rs builds C): + such a URL is an OSError naming the build feature.""" + for url, feature in [("s3://bucket/k.h5", "s3"), ("gs://b/k.h5", "gcs"), ("az://c/k.h5", "azure")]: + try: + clawhdf5.File(url) + except OSError as e: + if "feature" in str(e): + assert f"`{feature}`" in str(e), str(e) + else: + pytest.fail(f"{url} opened") + + +def test_https_needs_the_https_feature(): + """The default wheel has no TLS stack (rustls needs ring, which builds C): + an https URL is an OSError that names the build feature.""" + with pytest.raises(OSError) as e: + clawhdf5.File("https://127.0.0.1:1/x.h5") + msg = str(e.value) + # Built with `--features https` the error is the refused connection. + assert "https" in msg or "connect" in msg.lower() or "refused" in msg.lower(), msg + + +def test_a_changed_file_is_an_error_not_mixed_data(h5py, remote_file): + path, url, _ = remote_file + f = clawhdf5.File.open_url(url, block_size=1024) + first = f["grp/small"][...] + # Rewrite the file with other values: new ETag, same name. + time.sleep(0.01) + with h5py.File(path, "w") as g: + g.create_dataset("contig", data=np.zeros((400, 300))) + with pytest.raises(OSError, match="changed"): + f["contig"][...] + np.testing.assert_array_equal(first, np.arange(10, dtype=" before + assert took >= 0.2, took + assert progress["n"] > 1000, progress + # Held across a 0.2 s request, the spinner would stall that long. + assert progress["worst"] < 0.1, (progress, took) + + +def test_a_clawhdf5_written_file_reads_the_same_remotely(tmp_path, range_server): + path = tmp_path / "ours.h5" + data = np.arange(3000, dtype=" Promise` backed by `fetch` with a diff --git a/docs/known-issues.md b/docs/known-issues.md index e97f602..bf4ae74 100644 --- a/docs/known-issues.md +++ b/docs/known-issues.md @@ -844,9 +844,9 @@ cache, but: `read_*_zerocopy`) need the file in memory and answer `FormatError::ContiguousStorageRequired` otherwise; `File::as_bytes()` panics for such a file (`File::contiguous_bytes()` is the fallible form). - `LazyFile`, `MmapFile` and the Python and wasm bindings still read a - whole file (`h5rs` reads through `File::storage`, and takes URLs with its - `remote` feature). + `LazyFile`, `MmapFile` and the wasm bindings still read a whole file + (`h5rs` and the Python bindings read through `File::storage`, and take + URLs: `h5rs` with its `remote` feature, Python with `clawhdf5.File(url)`). - The file's length is read once, at open: a growing file (SWMR) is not followed (milestone M5). A remote file is pinned at open, so one that grows is `RemoteError::FileChanged`. @@ -862,9 +862,12 @@ cache, but: **Status:** open (added 2026-09-26, milestone M3 of `docs/design/range-reads.md`). -- **Python and the browser cannot open URLs yet.** `clawhdf5.File` (PyO3) - parses through `File::as_bytes`, which a remote file does not have; the - wasm reader's `openUrl` is milestone M4. +- **The browser cannot open URLs yet**: the wasm reader's `openUrl` is + milestone M4. Python can (`clawhdf5.File(url)`, since 2026-09-27), but + the default wheel reads plain `http://` only: `https://` needs a wheel + built with `--features https` (rustls with ring, which compiles C), and + `s3://`, `gs://`, `az://` the `s3`, `gcs`, `azure` features (aws-lc-rs). + The Python tests run against an in-process `http.server` only. - **The block size is fixed** (1 MiB unless `CacheConfig` says otherwise). The design's policy of using a paged file's page size as the block size is not implemented, and only the first block is read ahead. diff --git a/scripts/ci-test.sh b/scripts/ci-test.sh index 4f678ad..5731358 100755 --- a/scripts/ci-test.sh +++ b/scripts/ci-test.sh @@ -131,14 +131,17 @@ run_step "cargo clippy (h5rs remote)" cargo clippy \ # js-sys (clawhdf5-wasm's bindings to JavaScript) builds no C. # clawhdf5-remote is checked by default (plain HTTP) and with its # object-store feature, and h5rs with URL support (remote); the https -# (ring) and s3/gcs/azure (aws-lc-rs) features build C and are opt-in. +# (ring) and s3/gcs/azure (aws-lc-rs) features build C and are opt-in. The +# Python bindings (clawhdf5-py, remote reads over plain HTTP) are checked too: +# their https/s3/gcs/azure features are opt-in for the same reason. no_c_in_default_build() { local entry crate features found=0 for entry in clawhdf5-format clawhdf5-io clawhdf5-filters clawhdf5 \ clawhdf5-agent clawhdf5-ann clawhdf5-accel clawhdf5-netcdf4 clawhdf5-cli \ clawhdf5-tools \ clawhdf5-wasm \ - clawhdf5-remote clawhdf5-remote:object-store clawhdf5-tools:remote; do + clawhdf5-remote clawhdf5-remote:object-store clawhdf5-tools:remote \ + clawhdf5-py; do crate=${entry%%:*} features=() [ "$entry" != "$crate" ] && features=(--features "${entry#*:}")