Lazy remote files in the browser (M4), SWMR reader (M5), Python remote reads and editing #19
@@ -518,17 +518,31 @@ races the writer (a checksum that fails mid-flush) is retried, up to 100
|
||||
attempts as in libhdf5, and never returned torn.
|
||||
|
||||
```rust
|
||||
use std::time::{Duration, Instant};
|
||||
|
||||
let file = clawhdf5::File::open_swmr("live.h5")?;
|
||||
let mut ds = file.dataset("samples")?;
|
||||
while file.swmr_writer_active()? {
|
||||
let (mut seen, mut last_growth) = (0, Instant::now());
|
||||
// Stop when the writer closes the file, or when the dataset has not grown
|
||||
// for a minute: a writer that crashed or was killed never clears the
|
||||
// SWMR-write flag, so `swmr_writer_active()` alone can stay true forever.
|
||||
while file.swmr_writer_active()? && last_growth.elapsed() < Duration::from_secs(60) {
|
||||
ds.refresh()?;
|
||||
let n = ds.shape()?[0];
|
||||
// read the new rows ...
|
||||
std::thread::sleep(std::time::Duration::from_millis(100));
|
||||
if n > seen {
|
||||
// read rows seen..n ...
|
||||
(seen, last_growth) = (n, Instant::now());
|
||||
}
|
||||
std::thread::sleep(Duration::from_millis(100));
|
||||
}
|
||||
ds.refresh()?; // the final extent
|
||||
```
|
||||
|
||||
`swmr_writer_active()` reads the superblock's SWMR-write flag, which libhdf5
|
||||
clears only when the writer closes the file; a file whose writer died keeps
|
||||
it set (as the mid-write copy in `tests/fixtures/swmr_mid_write.h5` does),
|
||||
so a follower needs its own stop condition, like the idle timeout above.
|
||||
|
||||
Design and limits: [docs/design/swmr.md](docs/design/swmr.md).
|
||||
|
||||
### Python
|
||||
|
||||
@@ -616,10 +616,34 @@ impl File {
|
||||
self.swmr.retries()
|
||||
}
|
||||
|
||||
/// Whether a SWMR writer has the file open now: the superblock is read
|
||||
/// again and its SWMR-write flag returned. libhdf5 clears the flag when
|
||||
/// the writer closes the file, so a reader can stop following it then
|
||||
/// (and one last [`Dataset::refresh`] sees the final extents).
|
||||
/// Whether a SWMR writer has the file open now, as far as the file
|
||||
/// says: the superblock is read again and its SWMR-write flag returned.
|
||||
/// libhdf5 clears the flag when the writer closes the file, so a reader
|
||||
/// can stop following it then (and one last [`Dataset::refresh`] sees
|
||||
/// the final extents). A writer that crashed or was killed never clears
|
||||
/// it, so the flag alone is not a stop condition: give the loop another
|
||||
/// one, such as a time without growth.
|
||||
///
|
||||
/// ```no_run
|
||||
/// # fn main() -> Result<(), clawhdf5::Error> {
|
||||
/// use std::time::{Duration, Instant};
|
||||
///
|
||||
/// let file = clawhdf5::File::open_swmr("live.h5")?;
|
||||
/// let mut ds = file.dataset("samples")?;
|
||||
/// let (mut seen, mut last_growth) = (0, Instant::now());
|
||||
/// while file.swmr_writer_active()? && last_growth.elapsed() < Duration::from_secs(60) {
|
||||
/// ds.refresh()?;
|
||||
/// let n = ds.shape()?[0];
|
||||
/// if n > seen {
|
||||
/// // read rows seen..n ...
|
||||
/// (seen, last_growth) = (n, Instant::now());
|
||||
/// }
|
||||
/// std::thread::sleep(Duration::from_millis(100));
|
||||
/// }
|
||||
/// ds.refresh()?; // the final extent
|
||||
/// # Ok(())
|
||||
/// # }
|
||||
/// ```
|
||||
pub fn swmr_writer_active(&self) -> Result<bool, Error> {
|
||||
self.retry(|| {
|
||||
let sb = Superblock::parse_in(&self.data, 0)?;
|
||||
|
||||
@@ -121,6 +121,9 @@ writer keeps appending. What the format and the library guarantee:
|
||||
the index entry, only appends), as for libhdf5's reader.
|
||||
5. **Writer state.** `File::swmr_writer_active()` reads the superblock
|
||||
flags again, so a reader can tell when the writer has closed the file.
|
||||
A writer that crashed or was killed never clears the flag, so a reader
|
||||
that follows a file needs a second stop condition (the README example
|
||||
stops after a minute without growth).
|
||||
|
||||
What stays out: SWMR writing, VFD SWMR (HDF5 1.13's page-buffer protocol,
|
||||
not in 1.14 or 2.0), remote SWMR, refresh of groups/attributes (a SWMR
|
||||
|
||||
Reference in New Issue
Block a user