Lazy remote files in the browser (M4), SWMR reader (M5), Python remote reads and editing #19

Merged
osobh merged 33 commits from feat/p3-wasm-swmr-python into main 2026-09-27 14:47:26 +00:00
3 changed files with 48 additions and 7 deletions
Showing only changes of commit 8ef7e80473 - Show all commits
+17 -3
View File
@@ -518,17 +518,31 @@ races the writer (a checksum that fails mid-flush) is retried, up to 100
attempts as in libhdf5, and never returned torn.
```rust
use std::time::{Duration, Instant};
let file = clawhdf5::File::open_swmr("live.h5")?;
let mut ds = file.dataset("samples")?;
while file.swmr_writer_active()? {
let (mut seen, mut last_growth) = (0, Instant::now());
// Stop when the writer closes the file, or when the dataset has not grown
// for a minute: a writer that crashed or was killed never clears the
// SWMR-write flag, so `swmr_writer_active()` alone can stay true forever.
while file.swmr_writer_active()? && last_growth.elapsed() < Duration::from_secs(60) {
ds.refresh()?;
let n = ds.shape()?[0];
// read the new rows ...
std::thread::sleep(std::time::Duration::from_millis(100));
if n > seen {
// read rows seen..n ...
(seen, last_growth) = (n, Instant::now());
}
std::thread::sleep(Duration::from_millis(100));
}
ds.refresh()?; // the final extent
```
`swmr_writer_active()` reads the superblock's SWMR-write flag, which libhdf5
clears only when the writer closes the file; a file whose writer died keeps
it set (as the mid-write copy in `tests/fixtures/swmr_mid_write.h5` does),
so a follower needs its own stop condition, like the idle timeout above.
Design and limits: [docs/design/swmr.md](docs/design/swmr.md).
### Python
+28 -4
View File
@@ -616,10 +616,34 @@ impl File {
self.swmr.retries()
}
/// Whether a SWMR writer has the file open now: the superblock is read
/// again and its SWMR-write flag returned. libhdf5 clears the flag when
/// the writer closes the file, so a reader can stop following it then
/// (and one last [`Dataset::refresh`] sees the final extents).
/// Whether a SWMR writer has the file open now, as far as the file
/// says: the superblock is read again and its SWMR-write flag returned.
/// libhdf5 clears the flag when the writer closes the file, so a reader
/// can stop following it then (and one last [`Dataset::refresh`] sees
/// the final extents). A writer that crashed or was killed never clears
/// it, so the flag alone is not a stop condition: give the loop another
/// one, such as a time without growth.
///
/// ```no_run
/// # fn main() -> Result<(), clawhdf5::Error> {
/// use std::time::{Duration, Instant};
///
/// let file = clawhdf5::File::open_swmr("live.h5")?;
/// let mut ds = file.dataset("samples")?;
/// let (mut seen, mut last_growth) = (0, Instant::now());
/// while file.swmr_writer_active()? && last_growth.elapsed() < Duration::from_secs(60) {
/// ds.refresh()?;
/// let n = ds.shape()?[0];
/// if n > seen {
/// // read rows seen..n ...
/// (seen, last_growth) = (n, Instant::now());
/// }
/// std::thread::sleep(Duration::from_millis(100));
/// }
/// ds.refresh()?; // the final extent
/// # Ok(())
/// # }
/// ```
pub fn swmr_writer_active(&self) -> Result<bool, Error> {
self.retry(|| {
let sb = Superblock::parse_in(&self.data, 0)?;
+3
View File
@@ -121,6 +121,9 @@ writer keeps appending. What the format and the library guarantee:
the index entry, only appends), as for libhdf5's reader.
5. **Writer state.** `File::swmr_writer_active()` reads the superblock
flags again, so a reader can tell when the writer has closed the file.
A writer that crashed or was killed never clears the flag, so a reader
that follows a file needs a second stop condition (the README example
stops after a minute without growth).
What stays out: SWMR writing, VFD SWMR (HDF5 1.13's page-buffer protocol,
not in 1.14 or 2.0), remote SWMR, refresh of groups/attributes (a SWMR