py: remote files (clawhdf5.File(url), File.open_url) through File::storage()
The Python bindings could not open a remote file: they parsed through File::as_bytes() in eight places (path lookups, object headers, dataspaces, attributes, group listings, the global heap of variable-length data), which a storage-backed file does not have. - Every object of a File now shares one handle (src/handle.rs) that runs all file access, metadata included, with the GIL released and parses through File::storage() and the clawhdf5_format *_in functions. Local files take the same path (their storage is the mmap). - clawhdf5.File(url) opens any scheme://... through clawhdf5_remote::storage_for_url (read-only; another mode is a ValueError). File.open_url(url, **options) takes the cache and HTTP options (block_size, cache_size, headers, retries, timeout, allow_full_download, max_full_download, require_validator, max_redirects, max_parallel); File.remote_stats gives the block cache's counters. - Default build: plain HTTP only, no C. https (rustls/ring) and s3/gcs/azure (aws-lc-rs) are opt-in features of clawhdf5-py, and ci-test.sh's no-C check now covers the crate. - A failed storage read (network error, file changed on the server) is an OSError, never KeyError/ValueError and never data; `key in group` raises it instead of answering False. Tests: the read-vs-h5py suite runs locally and over HTTP (1 MiB and 1 KiB blocks) against a range-capable http.server in the test process (conftest.RangeServer); test_remote.py covers request counts, cache hits, a server without Range support, a changed file, a server that hangs up, 16 threads, and a spinning thread that keeps running while a read waits on 0.2 s requests. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
@@ -178,15 +178,32 @@ def _write_fixture(h5py, path):
|
||||
g.attrs["depth"] = np.int8(3)
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def pair(h5py, tmp_path_factory):
|
||||
path = str(tmp_path_factory.mktemp("h5") / "fixture.h5")
|
||||
@pytest.fixture(scope="module", params=["local", "http", "http-1k-blocks"])
|
||||
def pair(request, h5py, tmp_path_factory):
|
||||
"""The fixture file through h5py and through clawhdf5: opened locally,
|
||||
and over HTTP range requests (a local server in this process) with the
|
||||
default 1 MiB blocks and with 1 KiB blocks, so every structure is read
|
||||
through many small ranges."""
|
||||
from conftest import RangeServer
|
||||
|
||||
root = tmp_path_factory.mktemp("h5")
|
||||
path = str(root / "fixture.h5")
|
||||
_write_fixture(h5py, path)
|
||||
theirs = h5py.File(path, "r")
|
||||
ours = clawhdf5.File(path, "r")
|
||||
server = None
|
||||
if request.param == "local":
|
||||
ours = clawhdf5.File(path, "r")
|
||||
else:
|
||||
server = RangeServer(root)
|
||||
if request.param == "http":
|
||||
ours = clawhdf5.File(server.url("fixture.h5"))
|
||||
else:
|
||||
ours = clawhdf5.File.open_url(server.url("fixture.h5"), block_size=1024)
|
||||
yield ours, theirs, path
|
||||
theirs.close()
|
||||
ours.close()
|
||||
if server is not None:
|
||||
server.close()
|
||||
|
||||
|
||||
def _all_datasets(h5py, f):
|
||||
|
||||
Reference in New Issue
Block a user