docs: SWMR reader (range-read M5) in CHANGELOG, README, known issues and designs
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
@@ -101,6 +101,10 @@ breaking change, are in [CHANGELOG.md](CHANGELOG.md).
|
||||
HTTP server (or in S3/GCS/Azure, opt-in) by range requests through a
|
||||
block cache, without downloading it; `h5rs` takes URLs with its `remote`
|
||||
feature. See [Reading remote files](#reading-remote-files).
|
||||
- `File::open_swmr` follows a file an h5py/libhdf5 SWMR writer is still
|
||||
appending to (`Dataset::refresh`, bounded retries); copies of such files
|
||||
taken mid-write read with every open path. See
|
||||
[Following a file a SWMR writer is appending to](#following-a-file-a-swmr-writer-is-appending-to).
|
||||
|
||||
**Tooling**
|
||||
- CI now runs the h5py/netCDF4 interop suites for real (they had been skipping
|
||||
@@ -504,6 +508,29 @@ built with `--features remote` takes the same URLs:
|
||||
`https://` is the `https` feature (rustls with ring, which compiles C).
|
||||
Limits are in [known issues](docs/known-issues.md).
|
||||
|
||||
### Following a file a SWMR writer is appending to
|
||||
|
||||
`File::open_swmr` reads a file that a libhdf5 writer in SWMR mode (h5py
|
||||
`f.swmr_mode = True`) is still appending to, as h5py's
|
||||
`File(path, "r", swmr=True)` does: `Dataset::refresh()` picks up the new
|
||||
extent, every read reads the chunk index as it is now, and a read that
|
||||
races the writer (a checksum that fails mid-flush) is retried, up to 100
|
||||
attempts as in libhdf5, and never returned torn.
|
||||
|
||||
```rust
|
||||
let file = clawhdf5::File::open_swmr("live.h5")?;
|
||||
let mut ds = file.dataset("samples")?;
|
||||
while file.swmr_writer_active()? {
|
||||
ds.refresh()?;
|
||||
let n = ds.shape()?[0];
|
||||
// read the new rows ...
|
||||
std::thread::sleep(std::time::Duration::from_millis(100));
|
||||
}
|
||||
ds.refresh()?; // the final extent
|
||||
```
|
||||
|
||||
Design and limits: [docs/design/swmr.md](docs/design/swmr.md).
|
||||
|
||||
### Python
|
||||
|
||||
`crates/clawhdf5-py` is a Python package (PyO3 + numpy) that reads HDF5 with
|
||||
|
||||
Reference in New Issue
Block a user