py: remote files (clawhdf5.File(url), File.open_url) through File::storage()
The Python bindings could not open a remote file: they parsed through File::as_bytes() in eight places (path lookups, object headers, dataspaces, attributes, group listings, the global heap of variable-length data), which a storage-backed file does not have. - Every object of a File now shares one handle (src/handle.rs) that runs all file access, metadata included, with the GIL released and parses through File::storage() and the clawhdf5_format *_in functions. Local files take the same path (their storage is the mmap). - clawhdf5.File(url) opens any scheme://... through clawhdf5_remote::storage_for_url (read-only; another mode is a ValueError). File.open_url(url, **options) takes the cache and HTTP options (block_size, cache_size, headers, retries, timeout, allow_full_download, max_full_download, require_validator, max_redirects, max_parallel); File.remote_stats gives the block cache's counters. - Default build: plain HTTP only, no C. https (rustls/ring) and s3/gcs/azure (aws-lc-rs) are opt-in features of clawhdf5-py, and ci-test.sh's no-C check now covers the crate. - A failed storage read (network error, file changed on the server) is an OSError, never KeyError/ValueError and never data; `key in group` raises it instead of answering False. Tests: the read-vs-h5py suite runs locally and over HTTP (1 MiB and 1 KiB blocks) against a range-capable http.server in the test process (conftest.RangeServer); test_remote.py covers request counts, cache hits, a server without Range support, a changed file, a server that hangs up, 16 threads, and a spinning thread that keeps running while a read waits on 0.2 s requests. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
@@ -2,6 +2,36 @@
|
||||
|
||||
## Unreleased
|
||||
|
||||
### Python bindings: remote files (2026-09-27)
|
||||
- **`clawhdf5.File(url)`** opens `http://` URLs (and `https://`, `s3://`,
|
||||
`gs://`, `az://` in a wheel built with the `https`, `s3`, `gcs`,
|
||||
`azure` features) through `clawhdf5-remote`'s `open_url`: range requests
|
||||
through the block cache, the whole read API (groups, attributes, every
|
||||
dataset type and index the local reader handles). A URL is any
|
||||
`scheme://…`; a remote file is read-only (another mode is a
|
||||
`ValueError`). **`clawhdf5.File.open_url(url, **options)`** takes
|
||||
`block_size`, `cache_size`, `headers`, `retries`, `timeout`,
|
||||
`allow_full_download`, `max_full_download`, `require_validator`,
|
||||
`max_redirects` and `max_parallel`; `File.remote_stats` gives the block
|
||||
cache's counters. The default wheel builds plain HTTP only (no C: rustls
|
||||
needs ring, and the cloud clients aws-lc-rs), and `ci-test.sh`'s no-C
|
||||
check now covers `clawhdf5-py`.
|
||||
- **Every read parses through `File::storage()`** instead of
|
||||
`File::as_bytes()` (path lookups, object headers, dataspaces, attributes,
|
||||
group listings, variable-length data through the global heap), inside one
|
||||
shared file handle that releases the GIL for all file access, not only
|
||||
dataset reads: a read waiting on the network lets other Python threads
|
||||
run. A failed read of the storage (a network error, a file changed on
|
||||
the server) is an `OSError`, never a `KeyError`/`ValueError` and never
|
||||
data; `key in group` raises it instead of answering `False`.
|
||||
- Tests: the whole read-vs-h5py suite also runs over HTTP (default 1 MiB
|
||||
blocks and 1 KiB blocks) against a range-capable `http.server` in the
|
||||
test process, plus `tests/test_remote.py`: request counts of a small
|
||||
read, cache hits, a server without `Range` support (refused, or a
|
||||
whole download when allowed), a file changed on the server, a server
|
||||
that hangs up, 16 threads on one remote file, and a thread that keeps
|
||||
running while a read waits on 0.2 s requests.
|
||||
|
||||
### Range reads, milestone M3: remote files (2026-09-26)
|
||||
- **New crate `clawhdf5-remote`.** `open_url("http://host/file.h5")` gives
|
||||
a `clawhdf5::File` (through `File::open_storage`) that reads the file by
|
||||
|
||||
Reference in New Issue
Block a user