quantumclaw/clawhdf5
3
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

Merge branch 'feat/p3-m5-swmr-reader' into feat/p3-wasm-swmr-python

Browse Source
This commit is contained in:
osobh
2026-09-27 07:59:14 -05:00
parent a4c2aced55 2e906ffe3a
commit 527a1a7ac6
14 changed files with 2252 additions and 265 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CHANGELOG.md
+70
View File
@@ -2,6 +2,76 @@
## Unreleased
### Range reads, milestone M5: reading files a SWMR writer is appending to (2026-09-27)
Design: `docs/design/swmr.md`.
- **Fix: files with the SWMR-write flag were bounded by a stale end of
file** (since the end-of-file check of 2026-09-26, unreleased). A
libhdf5 SWMR writer (h5py `f.swmr_mode = True`) does not keep
the superblock's end-of-file address up to date; a copy h5py made of its
own file mid-write records 715 in a 6 030-byte file.
`Superblock::data_end` only ignored the recorded end when it lay past the
end of the file, so such a file listed, but every chunked read failed
("unexpected EOF: need 787 bytes, have 715") and `h5rs check` reported
its chunk indexes past the end of the file. For a v3 superblock with the
SWMR-write flag the data now ends at the end of the file, as libhdf5's
SWMR reader reads it (it skips the end-of-allocation check). Every open
path (`File::open`, `open_buffered`, `from_bytes`, `open_storage`,
`MmapFile`, `LazyFile`, `h5rs`) reads such a copy now.
- **`File::open_swmr(path)` / `File::open_storage_swmr(storage)`**: open a
file a SWMR writer may still be appending to, as libhdf5's SWMR reader
(h5py `File(path, "r", swmr=True)`) does. The file is read with
positioned reads (the new `clawhdf5::FileStorage`: `pread`/`seek_read`,
never mapped, `len()` the file's current length), reads are bounded by
the file's length at the time of each read, and the chunk cache is not
used (a cached chunk index would hide new chunks; a cached edge chunk
would read as fill where the writer has since written). A file open for
writing without SWMR is refused with `Error::Locked`, as libhdf5 refuses
it.
Only a file whose superblock has the SWMR-write flag when it is opened is
read this way; any other file (its writer has closed it) reads exactly as
`File::open` reads it, bounded by its recorded end of file, and
`is_swmr_read()` is `false`.
- **`Dataset::refresh()`** reads the dataset's object header again
(`H5Drefresh`, h5py `Dataset.refresh()`), so `shape()` and later reads
see the writer's appends; a handle keeps its extent until refreshed.
- **Bounded retries.** On a SWMR-read file, an operation (open, lookups
and listings, refresh, every dataset read including strings and
variable-length data, attributes, `File::decode_*`) that fails with an error a concurrent write can cause —
the failures libhdf5's SWMR reader retries: a checksum mismatch, a read
past the file's current end, an object header prefix that does not
decode — is run again from the start, up to
`File::swmr_read_attempts()` times (`SWMR_READ_ATTEMPTS` = 100,
libhdf5's default metadata read attempts for SWMR readers,
`H5Pset_metadata_read_attempts`; `set_swmr_read_attempts` changes it),
pausing 1 µs doubling to 10 ms between attempts. Results come only from
an attempt in which every structure verified, so a torn read is at worst
an error, never data. Any other error is returned at once.
`File::swmr_retries()` counts the retries.
`File::swmr_writer_active()` reads the superblock flags again to tell
when the writer has closed the file.
- Tests (`crates/clawhdf5/tests/swmr_interop.rs`): the mid-write copy
(fixture `tests/fixtures/swmr_mid_write.h5`) through every open path and
against h5py's SWMR reader; garbled reads (a storage that corrupts the
next reads) retried, never returned, and given up after the attempts;
every read path (listings, attributes compact and dense, strings,
variable-length data; fixture `tests/fixtures/swmr_strings_attrs.h5`)
with each of its reads failing once in turn returns the same result;
and a live test: an h5py writer appends to a 1-D and a 2-D dataset with
one unlimited dimension (Extensible Array index, one of them gzip) and a
2-D dataset with two (v2 B-tree index) for 2 500 steps
(`CLAWHDF5_SWMR_STEPS`), flushing after each, while two Rust reader
threads refresh and read all three in a loop — every value must be the
one the writer wrote at its position, extents never shrink — beside
h5py's own SWMR reader doing the same checks; after the writer closes,
the live handle and a new `File::open` read exactly what h5py reads.
Run on tank, 2026-09-27, with h5py 3.16 / HDF5 2.0:
`CLAWHDF5_REQUIRE_INTEROP=1 CLAWHDF5_PYTHON=.venv/bin/python cargo test
-p clawhdf5 --test swmr_interop`; also at 20 000 steps in a release
build. A test with the chunk cache left on in live mode fails it.
- Not covered: SWMR writing, remote SWMR (`BlockCache` caches blocks and
`HttpStorage` pins the length), `MmapFile`/`LazyFile`, and refreshing
groups or attributes (a SWMR writer cannot add objects or attributes).
### Range reads, milestone M3: remote files (2026-09-26)
- **New crate `clawhdf5-remote`.** `open_url("http://host/file.h5")` gives
a `clawhdf5::File` (through `File::open_storage`) that reads the file by
README.md
+41
View File
@@ -101,6 +101,10 @@ breaking change, are in [CHANGELOG.md](CHANGELOG.md).
HTTP server (or in S3/GCS/Azure, opt-in) by range requests through a
block cache, without downloading it; `h5rs` takes URLs with its `remote`
feature. See [Reading remote files](#reading-remote-files).
- `File::open_swmr` follows a file an h5py/libhdf5 SWMR writer is still
appending to (`Dataset::refresh`, bounded retries); copies of such files
taken mid-write read with every open path. See
[Following a file a SWMR writer is appending to](#following-a-file-a-swmr-writer-is-appending-to).
**Tooling**
- CI now runs the h5py/netCDF4 interop suites for real (they had been skipping
@@ -504,6 +508,43 @@ built with `--features remote` takes the same URLs:
`https://` is the `https` feature (rustls with ring, which compiles C).
Limits are in [known issues](docs/known-issues.md).
### Following a file a SWMR writer is appending to
`File::open_swmr` reads a file that a libhdf5 writer in SWMR mode (h5py
`f.swmr_mode = True`) is still appending to, as h5py's
`File(path, "r", swmr=True)` does: `Dataset::refresh()` picks up the new
extent, every read reads the chunk index as it is now, and a read that
races the writer (a checksum that fails mid-flush) is retried, up to 100
attempts as in libhdf5, and never returned torn.
```rust
use std::time::{Duration, Instant};
let file = clawhdf5::File::open_swmr("live.h5")?;
let mut ds = file.dataset("samples")?;
let (mut seen, mut last_growth) = (0, Instant::now());
// Stop when the writer closes the file, or when the dataset has not grown
// for a minute: a writer that crashed or was killed never clears the
// SWMR-write flag, so `swmr_writer_active()` alone can stay true forever.
while file.swmr_writer_active()? && last_growth.elapsed() < Duration::from_secs(60) {
ds.refresh()?;
let n = ds.shape()?[0];
if n > seen {
// read rows seen..n ...
(seen, last_growth) = (n, Instant::now());
}
std::thread::sleep(Duration::from_millis(100));
}
ds.refresh()?; // the final extent
```
`swmr_writer_active()` reads the superblock's SWMR-write flag, which libhdf5
clears only when the writer closes the file; a file whose writer died keeps
it set (as the mid-write copy in `tests/fixtures/swmr_mid_write.h5` does),
so a follower needs its own stop condition, like the idle timeout above.
Design and limits: [docs/design/swmr.md](docs/design/swmr.md).
### Python
`crates/clawhdf5-py` is a Python package (PyO3 + numpy) that reads HDF5 with
crates/clawhdf5-format/src/attribute.rs
+53 -18
View File
@@ -576,7 +576,14 @@ pub fn find_attribute_in_file(
offset_size: u8,
length_size: u8,
) -> Result<Option<AttributeMessage>, FormatError> {
find_attribute_core(file_data, header, name, offset_size, length_size)
find_attribute_core(
file_data,
header,
name,
offset_size,
length_size,
&mut Vec::new(),
)
}
/// [`find_attribute_in_file`] over any [`Storage`] (see
@@ -594,16 +601,48 @@ pub fn find_attribute_in<S: Storage + ?Sized>(
) -> Result<Option<AttributeMessage>, FormatError> {
match file_data.as_contiguous() {
Some(all) => find_attribute_in_file(all, header, name, offset_size, length_size),
None => find_attribute_core(file_data, header, name, offset_size, length_size),
None => find_attribute_core(
file_data,
header,
name,
offset_size,
length_size,
&mut Vec::new(),
),
}
}
/// [`find_attribute_in`], also returning the errors of the attributes it
/// could not read on the way (which it leaves out rather than failing
/// the call): the attribute asked for may be one of them. A reader of a
/// file that is being written uses them to tell a read that raced the
/// writer from an absent attribute.
pub fn find_attribute_reporting_in<S: Storage + ?Sized>(
file_data: &S,
header: &ObjectHeader,
name: &str,
offset_size: u8,
length_size: u8,
) -> Result<(Option<AttributeMessage>, Vec<FormatError>), FormatError> {
let mut errors = Vec::new();
let found = find_attribute_core(
file_data,
header,
name,
offset_size,
length_size,
&mut errors,
)?;
Ok((found, errors))
}
fn find_attribute_core<S: Storage + ?Sized>(
file_data: &S,
header: &ObjectHeader,
name: &str,
offset_size: u8,
length_size: u8,
errors: &mut Vec<FormatError>,
) -> Result<Option<AttributeMessage>, FormatError> {
let attr_info = find_attribute_info(header, offset_size)?;
let dense = attr_info
@@ -612,12 +651,10 @@ fn find_attribute_core<S: Storage + ?Sized>(
let Some((fh_addr, btree_addr)) = dense else {
// Compact only (or dense storage without a name index, which a
// listing reports): as a listing finds it.
return Ok(
extract_attributes_tolerant_in(file_data, header, offset_size, length_size)?
.0
.into_iter()
.find(|a| a.name == name),
);
let (attrs, errs) =
extract_attributes_tolerant_in(file_data, header, offset_size, length_size)?;
errors.extend(errs);
return Ok(attrs.into_iter().find(|a| a.name == name));
};
let btree_hdr = BTreeV2Header::parse_in(
file_data,
@@ -627,12 +664,10 @@ fn find_attribute_core<S: Storage + ?Sized>(
)?;
let fh = FractalHeapHeader::parse_in(file_data, fh_addr, offset_size, length_size)?;
if btree_hdr.tree_type != ATTRIBUTE_NAME_INDEX || btree_hdr.record_size < 4 {
return Ok(
extract_attributes_tolerant_in(file_data, header, offset_size, length_size)?
.0
.into_iter()
.find(|a| a.name == name),
);
let (attrs, errs) =
extract_attributes_tolerant_in(file_data, header, offset_size, length_size)?;
errors.extend(errs);
return Ok(attrs.into_iter().find(|a| a.name == name));
}
// A listing has the compact attributes first.
@@ -671,10 +706,10 @@ fn find_attribute_core<S: Storage + ?Sized>(
AttributeMessage::parse_in_storage(&d, file_data, offset_size, length_size)
});
// One that cannot be read is left out, as from a listing.
if let Ok(attr) = attr
&& attr.name == name
{
return Ok(Some(attr));
match attr {
Ok(attr) if attr.name == name => return Ok(Some(attr)),
Ok(_) => {}
Err(e) => errors.push(e),
}
}
Ok(None)
crates/clawhdf5-format/src/superblock.rs
+19 -7
View File
@@ -116,22 +116,27 @@ impl Superblock {
/// [`FormatError::TruncatedFile`]. Bytes past that address are not part
/// of the file: libhdf5 fails any read of them ("addr overflow" /
/// "address plus size exceeds file eoa"), so a reader should parse only
/// the data up to the returned end. As libhdf5 does for a SWMR reader,
/// the check is skipped for a version-3 superblock whose writer is still
/// writing it in SWMR mode (it extends the file as it goes); the data
/// then ends at the end of the file.
/// the data up to the returned end.
///
/// A version-3 superblock with the SWMR-write flag set belongs to a file
/// a SWMR writer has open (or had, and did not close). That writer does
/// not keep the recorded end of file up to date — a copy taken mid-write
/// can record an end of a few hundred bytes in a file of tens of
/// kilobytes — and libhdf5's SWMR reader skips its end-of-allocation
/// check for every read (`H5FD_read`). For such a superblock the data
/// ends at the end of the file, whatever end it records.
///
/// When the superblock's recorded base address differs from where the
/// superblock actually is (a user block added or removed after the file
/// was written), libhdf5 moves the recorded end of file by the same
/// amount, and so does this.
pub fn data_end(&self, user_block: u64, file_len: u64) -> Result<u64, FormatError> {
if self.version >= 3 && self.is_swmr_write() {
return Ok(file_len.saturating_sub(user_block));
}
let eof =
i128::from(self.eof_address) - i128::from(self.base_address) + i128::from(user_block);
if eof < 0 || eof > i128::from(file_len) {
if self.version >= 3 && self.is_swmr_write() {
return Ok(file_len.saturating_sub(user_block));
}
return Err(FormatError::TruncatedFile {
stored_eof: u64::try_from(eof).unwrap_or(self.eof_address),
actual_len: file_len,
@@ -617,6 +622,13 @@ mod tests {
let mut swmr = Superblock::parse(&build_v2_bytes(8, 3), 0).unwrap();
swmr.consistency_flags = swmr_flags::WRITE_ACCESS | swmr_flags::SWMR_WRITE;
assert_eq!(swmr.data_end(0, 1000), Ok(1000));
// ... nor bounded by its recorded end, which the writer does not
// keep up to date (2048 here).
assert_eq!(swmr.data_end(0, 17_857), Ok(17_857));
assert_eq!(swmr.data_end(512, 17_857), Ok(17_345));
// Without the SWMR-write flag the recorded end bounds the data.
swmr.consistency_flags = swmr_flags::WRITE_ACCESS;
assert_eq!(swmr.data_end(0, 17_857), Ok(2048));
}
#[test]
crates/clawhdf5/src/lib.rs
+2
View File
@@ -47,6 +47,7 @@ pub mod lazy;
#[cfg(feature = "mmap")]
pub mod mmap_file;
pub mod reader;
mod swmr;
pub mod types;
pub mod vlen;
pub mod writer;
@@ -57,6 +58,7 @@ pub use lazy::{LazyDataset, LazyFile, LazyGroup};
#[cfg(feature = "mmap")]
pub use mmap_file::{MmapDataset, MmapFile, MmapGroup};
pub use reader::{Dataset, File, Group, SharedStorage, VdsResolver};
pub use swmr::{FileStorage, SWMR_READ_ATTEMPTS};
pub use types::{AttrValue, DType};
pub use vlen::VlenValue;
pub use writer::FileBuilder;
crates/clawhdf5/src/reader.rs
+575 -218
View File
File diff suppressed because it is too large Load Diff
crates/clawhdf5/src/swmr.rs
+362
View File
@@ -0,0 +1,362 @@
//! Reading files a SWMR writer is still appending to (see
//! `docs/design/swmr.md` and [`File::open_swmr`](crate::File::open_swmr)):
//! a [`FileStorage`] whose length grows with the file, and the bounded
//! retries of operations a concurrent write can make fail.
use std::borrow::Cow;
use std::cell::Cell;
use std::path::Path;
use std::sync::atomic::{AtomicU64, Ordering};
use std::time::Duration;
use clawhdf5_format::error::FormatError;
use clawhdf5_format::storage::Storage;
use crate::error::Error;
/// How many times a live file ([`File::open_swmr`](crate::File::open_swmr))
/// tries an operation that fails with an error a concurrent write can cause
/// before returning the error: 100, libhdf5's default number of metadata
/// read attempts for SWMR access (`H5Pset_metadata_read_attempts`).
pub const SWMR_READ_ATTEMPTS: u32 = 100;
/// Longest pause between two attempts.
const MAX_PAUSE: Duration = Duration::from_millis(10);
/// A local file read with positioned reads (`pread` on Unix, `seek_read` on
/// Windows), never mapped, whose [`Storage::len`] is the file's length at
/// the time of the call: a [`Storage`] for a file that another process is
/// appending to. A read past the end is short, as the trait allows.
#[derive(Debug)]
pub struct FileStorage {
file: std::fs::File,
#[cfg(not(any(unix, windows)))]
lock: std::sync::Mutex<()>,
}
impl FileStorage {
/// Open the file at `path` for reading.
pub fn open<P: AsRef<Path>>(path: P) -> std::io::Result<Self> {
Ok(Self::new(std::fs::File::open(path)?))
}
/// A storage over an open file.
pub fn new(file: std::fs::File) -> Self {
Self {
file,
#[cfg(not(any(unix, windows)))]
lock: std::sync::Mutex::new(()),
}
}
/// Up to `buf.len()` bytes at `offset`; fewer only at the end of file.
fn read_into(&self, offset: u64, buf: &mut [u8]) -> std::io::Result<usize> {
let mut got = 0;
while got < buf.len() {
match self.read_once(offset + got as u64, &mut buf[got..]) {
Ok(0) => break,
Ok(n) => got += n,
Err(e) if e.kind() == std::io::ErrorKind::Interrupted => {}
Err(e) => return Err(e),
}
}
Ok(got)
}
#[cfg(unix)]
fn read_once(&self, offset: u64, buf: &mut [u8]) -> std::io::Result<usize> {
std::os::unix::fs::FileExt::read_at(&self.file, buf, offset)
}
#[cfg(windows)]
fn read_once(&self, offset: u64, buf: &mut [u8]) -> std::io::Result<usize> {
std::os::windows::fs::FileExt::seek_read(&self.file, buf, offset)
}
#[cfg(not(any(unix, windows)))]
fn read_once(&self, offset: u64, buf: &mut [u8]) -> std::io::Result<usize> {
use std::io::{Read, Seek, SeekFrom};
let _guard = self.lock.lock().unwrap_or_else(|p| p.into_inner());
let mut f = &self.file;
f.seek(SeekFrom::Start(offset))?;
f.read(buf)
}
}
impl Storage for FileStorage {
fn read_at(&self, offset: u64, len: usize) -> Result<Cow<'_, [u8]>, FormatError> {
// Never allocate more than the file holds, whatever a (possibly
// hostile) size field asked for.
let avail = self.len().saturating_sub(offset);
let len = usize::try_from(avail).map_or(len, |a| a.min(len));
let mut buf = vec![0u8; len];
let got = self
.read_into(offset, &mut buf)
.map_err(|e| FormatError::Storage(format!("read of {len} bytes at {offset}: {e}")))?;
buf.truncate(got);
Ok(Cow::Owned(buf))
}
fn len(&self) -> u64 {
self.file.metadata().map_or(0, |m| m.len())
}
}
/// Whether `e` can be caused by reading a structure while a SWMR writer
/// rewrites it or has not yet written it, so that the operation is worth
/// running again (see [`is_transient_format`]).
pub(crate) fn is_transient(e: &Error) -> bool {
match e {
Error::Format(f) => is_transient_format(f),
Error::Io(io) => io.kind() == std::io::ErrorKind::UnexpectedEof,
_ => false,
}
}
/// The failures libhdf5's SWMR reader retries, and nothing else. libhdf5
/// (`H5C__load_entry`) reads a metadata structure again when its checksum
/// fails, and when the prefix it decodes before the checksum to learn the
/// structure's size does not decode (for an object header, its signature
/// and version: a header whose every byte is garbled fails there). A read
/// past the file's current end is short here; libhdf5 reads zeros there,
/// which then fail the checksum. So:
///
/// - [`FormatError::ChecksumMismatch`], of any checksummed structure;
/// - [`FormatError::UnexpectedEof`], a read past the current end;
/// - [`FormatError::InvalidObjectHeaderSignature`] and
/// [`FormatError::InvalidObjectHeaderVersion`], the object header prefix.
///
/// Every other error — an unsupported version or message, a file that is not
/// HDF5, a structure that is corrupt behind a valid checksum — is returned
/// at once: a concurrent write does not cause it, and retrying it only
/// costs up to a second of pauses.
pub(crate) fn is_transient_format(e: &FormatError) -> bool {
use FormatError as F;
matches!(
e,
F::ChecksumMismatch { .. }
| F::UnexpectedEof { .. }
| F::InvalidObjectHeaderSignature
| F::InvalidObjectHeaderVersion(_)
)
}
thread_local! {
/// Set while this thread runs an operation under [`retry`], so an
/// operation made of retried operations retries as a whole, not each
/// part up to the limit.
static RETRYING: Cell<bool> = const { Cell::new(false) };
}
/// A live file's retry policy: attempts per operation, and a count of the
/// retries made.
#[derive(Debug)]
pub(crate) struct Retries {
pub(crate) attempts: u32,
retried: AtomicU64,
}
impl Default for Retries {
fn default() -> Self {
Self {
attempts: SWMR_READ_ATTEMPTS,
retried: AtomicU64::new(0),
}
}
}
impl Retries {
/// [`retry`] with this policy, counting the retries.
pub(crate) fn retry<T>(&self, op: impl FnMut() -> Result<T, Error>) -> Result<T, Error> {
retry(self.attempts, &self.retried, op)
}
/// Retries made so far.
pub(crate) fn retries(&self) -> u64 {
self.retried.load(Ordering::Relaxed)
}
}
/// Run `op` up to `attempts` times while it fails with a transient error
/// ([`is_transient`]), pausing 1 µs, 2 µs, 4 µs, … up to 10 ms between
/// attempts, and adding each retry to `retried`; return its first success
/// or last error. Inside another `retry` on the same thread, `op` runs
/// once.
fn retry<T>(
attempts: u32,
retried: &AtomicU64,
mut op: impl FnMut() -> Result<T, Error>,
) -> Result<T, Error> {
if RETRYING.with(Cell::get) {
return op();
}
struct Reset;
impl Drop for Reset {
fn drop(&mut self) {
RETRYING.with(|r| r.set(false));
}
}
RETRYING.with(|r| r.set(true));
let _reset = Reset;
let mut pause = Duration::from_micros(1);
let mut attempt = 1;
loop {
match op() {
Err(e) if attempt < attempts && is_transient(&e) => {
retried.fetch_add(1, Ordering::Relaxed);
std::thread::sleep(pause);
pause = (pause * 2).min(MAX_PAUSE);
attempt += 1;
}
result => return result,
}
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn file_storage_reads_what_the_file_holds_now() {
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("grow.bin");
std::fs::write(&path, b"hello").unwrap();
let s = FileStorage::open(&path).unwrap();
assert_eq!(s.len(), 5);
assert_eq!(&*s.read_at(1, 3).unwrap(), b"ell");
assert_eq!(&*s.read_at(3, 10).unwrap(), b"lo");
assert!(s.read_at(9, 4).unwrap().is_empty());
// The file grows after the storage was opened.
use std::io::Write;
std::fs::OpenOptions::new()
.append(true)
.open(&path)
.unwrap()
.write_all(b", world")
.unwrap();
assert_eq!(s.len(), 12);
assert_eq!(&*s.read_at(3, 100).unwrap(), b"lo, world");
}
#[test]
fn permanent_errors_are_returned_at_once() {
// Errors a concurrent write does not cause: none is retried, so
// none costs the pauses (about 0.9 s for 100 attempts).
let permanent = [
FormatError::SignatureNotFound,
FormatError::UnsupportedVersion(9),
FormatError::UnsupportedMessage(0x99),
FormatError::TruncatedFile {
stored_eof: 10,
actual_len: 5,
},
FormatError::InvalidDatatypeClass(15),
FormatError::InvalidLayoutVersion(9),
FormatError::InvalidBTreeSignature,
FormatError::ChunkedReadError("x".into()),
FormatError::DecompressionError("x".into()),
FormatError::Fletcher32Mismatch {
expected: 1,
computed: 2,
},
];
let n = AtomicU64::new(0);
let started = std::time::Instant::now();
for e in permanent {
assert!(!is_transient_format(&e), "{e:?}");
let mut calls = 0;
let r: Result<(), Error> = retry(SWMR_READ_ATTEMPTS, &n, || {
calls += 1;
Err(Error::Format(e.clone()))
});
assert!(r.is_err());
assert_eq!(calls, 1, "{e:?}");
}
assert_eq!(n.load(Ordering::Relaxed), 0);
assert!(started.elapsed() < Duration::from_millis(50));
assert!(!is_transient(&Error::Io(std::io::Error::other("x"))));
// The transient ones are retried to the limit.
for e in [
FormatError::ChecksumMismatch {
expected: 1,
computed: 2,
},
FormatError::UnexpectedEof {
expected: 8,
available: 0,
},
FormatError::InvalidObjectHeaderSignature,
FormatError::InvalidObjectHeaderVersion(0x4f),
] {
let mut calls = 0;
let _: Result<(), Error> = retry(3, &n, || {
calls += 1;
Err(Error::Format(e.clone()))
});
assert_eq!(calls, 3, "{e:?}");
}
assert!(is_transient(&Error::Io(
std::io::ErrorKind::UnexpectedEof.into()
)));
}
#[test]
fn retry_runs_again_only_for_transient_errors() {
let n = AtomicU64::new(0);
let mut calls = 0;
let r: Result<u32, Error> = retry(5, &n, || {
calls += 1;
if calls < 3 {
Err(Error::Format(FormatError::ChecksumMismatch {
expected: 1,
computed: 2,
}))
} else {
Ok(7)
}
});
assert_eq!(r.unwrap(), 7);
assert_eq!(calls, 3);
assert_eq!(n.load(Ordering::Relaxed), 2);
// Gives up after `attempts`.
let mut calls = 0;
let r: Result<(), Error> = retry(4, &n, || {
calls += 1;
Err(Error::Format(FormatError::UnexpectedEof {
expected: 8,
available: 0,
}))
});
assert!(r.is_err());
assert_eq!(calls, 4);
assert_eq!(n.load(Ordering::Relaxed), 5);
// A permanent error is returned at once.
let mut calls = 0;
let r: Result<(), Error> = retry(4, &n, || {
calls += 1;
Err(Error::Format(FormatError::UnsupportedFilter(999)))
});
assert!(r.is_err());
assert_eq!(calls, 1);
assert_eq!(n.load(Ordering::Relaxed), 5);
// Nested: the inner operation runs once per outer attempt.
let mut inner = 0;
let mut outer = 0;
let r: Result<(), Error> = retry(3, &n, || {
outer += 1;
retry(3, &n, || {
inner += 1;
Err(Error::Format(FormatError::InvalidObjectHeaderVersion(0x76)))
})
});
assert!(r.is_err());
assert_eq!((outer, inner), (3, 3));
// The flag is reset afterwards.
assert!(!RETRYING.with(Cell::get));
}
}
crates/clawhdf5/src/types.rs
+66 -21
View File
@@ -170,16 +170,38 @@ pub(crate) fn read_attrs<S: clawhdf5_format::storage::Storage + ?Sized>(
),
crate::Error,
> {
read_attrs_reporting(file_data, header, offset_size, length_size)
.map(|(attrs, errors, _)| (attrs, errors))
}
/// What [`read_attrs_reporting`] returns: the attributes, the errors of
/// those left out, and the errors behind values returned as
/// [`AttrValue::Raw`].
pub(crate) type AttrsReport = (
HashMap<String, AttrValue>,
Vec<clawhdf5_format::error::FormatError>,
Vec<clawhdf5_format::error::FormatError>,
);
/// [`read_attrs`], also returning the errors of reading the file for a
/// value (a variable-length string's global heap) that was returned as
/// [`AttrValue::Raw`] instead: a live file retries on them (see
/// `File::open_swmr`).
pub(crate) fn read_attrs_reporting<S: clawhdf5_format::storage::Storage + ?Sized>(
file_data: &S,
header: &clawhdf5_format::object_header::ObjectHeader,
offset_size: u8,
length_size: u8,
) -> Result<AttrsReport, crate::Error> {
let (msgs, errors) = clawhdf5_format::attribute::extract_attributes_tolerant_in(
file_data,
header,
offset_size,
length_size,
)?;
Ok((
attrs_to_map(&msgs, file_data, offset_size, length_size),
errors,
))
let mut read_errors = Vec::new();
let map = attrs_to_map_reporting(&msgs, file_data, offset_size, length_size, &mut read_errors);
Ok((map, errors, read_errors))
}
/// The attribute called `name` on the object with header `header`, decoded
@@ -192,30 +214,50 @@ pub(crate) fn read_attr<S: clawhdf5_format::storage::Storage + ?Sized>(
offset_size: u8,
length_size: u8,
) -> Result<Option<AttrValue>, crate::Error> {
let Some(msg) = clawhdf5_format::attribute::find_attribute_in(
read_attr_reporting(file_data, header, name, offset_size, length_size).map(|(v, _)| v)
}
/// [`read_attr`], also returning the errors of the attributes it could not
/// read on the way (the one asked for may be among them) and of reading the
/// file for its value, if one made it [`AttrValue::Raw`] (see
/// [`read_attrs_reporting`]).
pub(crate) fn read_attr_reporting<S: clawhdf5_format::storage::Storage + ?Sized>(
file_data: &S,
header: &clawhdf5_format::object_header::ObjectHeader,
name: &str,
offset_size: u8,
length_size: u8,
) -> Result<(Option<AttrValue>, Vec<clawhdf5_format::error::FormatError>), crate::Error> {
let (found, mut read_errors) = clawhdf5_format::attribute::find_attribute_reporting_in(
file_data,
header,
name,
offset_size,
length_size,
)?
else {
return Ok(None);
)?;
let Some(msg) = found else {
return Ok((None, read_errors));
};
Ok(attrs_to_map(
let value = attrs_to_map_reporting(
std::slice::from_ref(&msg),
file_data,
offset_size,
length_size,
&mut read_errors,
)
.remove(name))
.remove(name);
Ok((value, read_errors))
}
pub(crate) fn attrs_to_map<S: clawhdf5_format::storage::Storage + ?Sized>(
/// The attributes `attrs` by name, decoded; an error reading the file for
/// a value that is returned as [`AttrValue::Raw`] because of it is added
/// to `read_errors`.
pub(crate) fn attrs_to_map_reporting<S: clawhdf5_format::storage::Storage + ?Sized>(
attrs: &[clawhdf5_format::attribute::AttributeMessage],
file_data: &S,
offset_size: u8,
length_size: u8,
read_errors: &mut Vec<clawhdf5_format::error::FormatError>,
) -> HashMap<String, AttrValue> {
let mut map = HashMap::new();
for attr in attrs {
@@ -224,13 +266,11 @@ pub(crate) fn attrs_to_map<S: clawhdf5_format::storage::Storage + ?Sized>(
// verbatim as `AttrValue::Raw` rather than dropped — a partial
// attribute list with no indication anything is missing is worse than
// an undecoded value.
let val =
decode_attr_value(attr, file_data, offset_size, length_size).unwrap_or_else(|| {
AttrValue::Raw {
datatype: attr.datatype.clone(),
shape: attr.dataspace.dimensions.clone(),
data: attr.raw_data.clone(),
}
let val = decode_attr_value(attr, file_data, offset_size, length_size, read_errors)
.unwrap_or_else(|| AttrValue::Raw {
datatype: attr.datatype.clone(),
shape: attr.dataspace.dimensions.clone(),
data: attr.raw_data.clone(),
});
map.insert(attr.name.clone(), val);
}
@@ -271,6 +311,7 @@ fn decode_attr_value<S: clawhdf5_format::storage::Storage + ?Sized>(
file_data: &S,
offset_size: u8,
length_size: u8,
read_errors: &mut Vec<clawhdf5_format::error::FormatError>,
) -> Option<AttrValue> {
use clawhdf5_format::datatype::Datatype;
@@ -310,9 +351,13 @@ fn decode_attr_value<S: clawhdf5_format::storage::Storage + ?Sized>(
Datatype::VariableLength {
is_string: true, ..
} => {
let strings = attr
.read_vl_strings_in(file_data, offset_size, length_size)
.ok()?;
let strings = match attr.read_vl_strings_in(file_data, offset_size, length_size) {
Ok(strings) => strings,
Err(e) => {
read_errors.push(e);
return None;
}
};
if strings.len() == 1 {
Some(AttrValue::String(strings[0].clone()))
} else {
crates/clawhdf5/tests/fixtures/swmr_mid_write.h5
Vendored
BIN
View File
Binary file not shown.
crates/clawhdf5/tests/fixtures/swmr_strings_attrs.h5
Vendored
BIN
View File
Binary file not shown.
crates/clawhdf5/tests/swmr_interop.rs
+847
View File
@@ -0,0 +1,847 @@
//! Files a libhdf5 SWMR writer (h5py `f.swmr_mode = True`) has open: a copy
//! taken mid-write (fixture), and a live file appended to by an h5py writer
//! process while clawhdf5 and h5py's own SWMR reader read it (see
//! `docs/design/swmr.md`).
//!
//! The live tests need python3 with h5py; they are skipped without it,
//! unless `CLAWHDF5_REQUIRE_INTEROP=1`.
use std::io::{BufRead, BufReader, Read};
use std::path::{Path, PathBuf};
use std::process::{Command, Stdio};
use std::sync::Arc;
use std::sync::atomic::{AtomicBool, AtomicU32, Ordering};
use clawhdf5::{File, Selection};
fn python() -> String {
std::env::var("CLAWHDF5_PYTHON").unwrap_or_else(|_| "python3".to_string())
}
fn interop_required() -> bool {
std::env::var("CLAWHDF5_REQUIRE_INTEROP").is_ok_and(|v| v == "1")
}
fn python_available() -> bool {
Command::new(python())
.args(["-c", "import h5py, numpy"])
.output()
.map(|o| o.status.success())
.unwrap_or(false)
}
macro_rules! skip_if_no_python {
() => {
if !python_available() {
assert!(
!interop_required(),
"CLAWHDF5_REQUIRE_INTEROP=1 but python3 with h5py is not available"
);
eprintln!("SKIP: python3 with h5py not available");
return;
}
};
}
fn fixture(name: &str) -> PathBuf {
Path::new(env!("CARGO_MANIFEST_DIR"))
.join("tests/fixtures")
.join(name)
}
/// `swmr_mid_write.h5`: a copy h5py 3.16 (HDF5 2.0) made of its own file
/// while writing it in SWMR mode, after 4 appends of 37 rows to
/// `/a` (int64, chunks of 100, no filter: `a[i] = i + 1`) and `/b` (float64
/// `(n, 4)`, chunks of 16 x 4, gzip: `b[i, j] = 10 i + j + 1`), each
/// followed by a flush. Its superblock (v3) still has the SWMR-write flag
/// set and records an end of file of 715 in a 6 030-byte file.
fn check_mid_write_copy(f: &File) {
let sb = f.superblock();
assert_eq!(sb.version, 3);
assert!(sb.is_swmr_write());
let a = f.dataset("a").unwrap();
assert_eq!(a.shape().unwrap(), vec![148]);
let want_a: Vec<i64> = (1..=148).collect();
assert_eq!(a.read_i64().unwrap(), want_a);
let b = f.dataset("b").unwrap();
assert_eq!(b.shape().unwrap(), vec![148, 4]);
let want_b: Vec<f64> = (0..148)
.flat_map(|i| (0..4).map(move |j| (10 * i + j + 1) as f64))
.collect();
assert_eq!(b.read_f64().unwrap(), want_b);
}
#[test]
fn a_copy_taken_mid_write_reads_past_its_recorded_end_of_file() {
let path = fixture("swmr_mid_write.h5");
// The recorded end of file (715) is far below the file's length; the
// chunk indexes and chunks lie past it. libhdf5's SWMR reader does not
// bound reads by it, and neither does any open path here.
check_mid_write_copy(&File::open(&path).unwrap());
check_mid_write_copy(&File::open_buffered(&path).unwrap());
check_mid_write_copy(&File::from_bytes(std::fs::read(&path).unwrap()).unwrap());
let want_a: Vec<i64> = (1..=148).collect();
let want_b: Vec<f64> = (0..148)
.flat_map(|i| (0..4).map(move |j| (10 * i + j + 1) as f64))
.collect();
let mm = clawhdf5::MmapFile::open(&path).unwrap();
assert_eq!(mm.dataset("a").unwrap().read_i64().unwrap(), want_a);
assert_eq!(mm.dataset("b").unwrap().read_f64().unwrap(), want_b);
let lazy = clawhdf5::LazyFile::open_mmap(&path).unwrap();
assert_eq!(lazy.dataset("a").unwrap().read_i64().unwrap(), want_a);
assert_eq!(lazy.dataset("b").unwrap().read_f64().unwrap(), want_b);
let storage = File::open_storage(Arc::new(std::fs::read(&path).unwrap())).unwrap();
check_mid_write_copy(&storage);
}
#[test]
fn a_copy_taken_mid_write_reads_as_h5py_swmr_reader_reads_it() {
skip_if_no_python!();
let path = fixture("swmr_mid_write.h5");
// libhdf5 refuses a non-SWMR open of this file ("file is already open
// for write"); its SWMR reader reads the values checked above.
let script = format!(
r#"
import h5py, numpy as np
with h5py.File("{p}", "r", swmr=True, locking=False) as f:
a = f["a"][()]
b = f["b"][()]
assert np.array_equal(a, np.arange(148) + 1), a
assert np.array_equal(b, (np.arange(148)[:, None] * 10 + np.arange(4) + 1).astype("f8")), b
print("ok")
"#,
p = path.display()
);
let out = Command::new(python())
.args(["-c", &script])
.output()
.unwrap();
assert!(
out.status.success(),
"h5py failed:\n{}",
String::from_utf8_lossy(&out.stderr)
);
check_mid_write_copy(&File::open(&path).unwrap());
}
#[test]
fn open_swmr_reads_the_mid_write_copy() {
let f = File::open_swmr(fixture("swmr_mid_write.h5")).unwrap();
assert!(f.is_swmr_read());
assert_eq!(f.swmr_read_attempts(), clawhdf5::SWMR_READ_ATTEMPTS);
// The copy still has the SWMR-write flag: as far as it says, its writer
// is still writing.
assert!(f.swmr_writer_active().unwrap());
check_mid_write_copy(&f);
let mut a = f.dataset("a").unwrap();
a.refresh().unwrap();
assert_eq!(a.shape().unwrap(), vec![148]);
assert_eq!(f.swmr_retries(), 0);
assert!(
!File::open(fixture("swmr_mid_write.h5"))
.unwrap()
.is_swmr_read()
);
}
/// The mid-write copy with its superblock's flags set to `flags` (and the
/// superblock checksum updated).
fn mid_write_copy_with_flags(flags: u8) -> Vec<u8> {
let mut bytes = std::fs::read(fixture("swmr_mid_write.h5")).unwrap();
// Superblock v3, 8-byte offsets: 12 bytes, 4 addresses, checksum.
bytes[11] = flags;
let sum = clawhdf5_format::checksum::jenkins_lookup3(&bytes[..44]);
bytes[44..48].copy_from_slice(&sum.to_le_bytes());
bytes
}
#[test]
fn open_swmr_refuses_a_file_open_for_writing_without_swmr() {
// libhdf5 refuses it too: "file is already open for write".
let bytes = mid_write_copy_with_flags(0x01);
let err = File::open_storage_swmr(Arc::new(bytes)).unwrap_err();
assert!(matches!(err, clawhdf5::Error::Locked(_)), "{err}");
// Closed (flags clear): opens, and no writer is active.
let f = File::open_storage_swmr(Arc::new(mid_write_copy_with_flags(0))).unwrap();
assert!(!f.swmr_writer_active().unwrap());
}
/// The outcome of reading every dataset of `f` in full, as text: the
/// values, or the error.
fn read_all(f: &File) -> Vec<String> {
["a", "b"]
.iter()
.map(|n| match f.dataset(n).and_then(|d| d.read_f64()) {
Ok(v) => format!("{n}: {} values", v.len()),
Err(e) => format!("{n}: error {e}"),
})
.collect()
}
#[test]
fn open_swmr_reads_a_file_without_the_swmr_write_flag_as_file_open_does() {
// The mid-write copy with its flags cleared: a closed file that
// records an end of file of 715 in 6 030 bytes, its chunk indexes past
// that end. File::open (and libhdf5's plain reader) refuse what lies
// past the recorded end; open_swmr must too, and must not retry (the
// file is not live). See the h5py test below for libhdf5's SWMR reader.
let bytes = mid_write_copy_with_flags(0);
let dir = tempfile::tempdir_in(env!("CARGO_TARGET_TMPDIR")).unwrap();
let path = dir.path().join("closed_short_eof.h5");
std::fs::write(&path, &bytes).unwrap();
let plain = File::open(&path).unwrap();
let swmr = File::open_swmr(&path).unwrap();
let swmr_storage = File::open_storage_swmr(Arc::new(bytes)).unwrap();
let want = read_all(&plain);
assert!(
want.iter().all(|r| r.contains("error")),
"File::open reads past the recorded end: {want:?}"
);
for f in [&swmr, &swmr_storage] {
assert!(!f.is_swmr_read());
assert!(!f.swmr_writer_active().unwrap());
let started = std::time::Instant::now();
assert_eq!(read_all(f), want);
assert!(started.elapsed() < std::time::Duration::from_millis(100));
assert_eq!(f.swmr_retries(), 0);
}
// The same bytes with the SWMR-write flag (and write access) set are
// read live, past the recorded end.
let live = File::open_storage_swmr(Arc::new(mid_write_copy_with_flags(0x05))).unwrap();
assert!(live.is_swmr_read());
check_mid_write_copy(&live);
}
#[test]
fn how_h5py_reads_the_closed_copy_past_its_recorded_end_of_file() {
skip_if_no_python!();
let dir = tempfile::tempdir_in(env!("CARGO_TARGET_TMPDIR")).unwrap();
let path = dir.path().join("closed_short_eof.h5");
std::fs::write(&path, mid_write_copy_with_flags(0)).unwrap();
// libhdf5's plain reader refuses `a` (its chunk index is past the
// recorded end of file), as File::open and open_swmr do above.
// libhdf5's SWMR reader skips its end-of-allocation check for every
// file it opens (H5FD_read), and reads it; it still refuses an object
// *header* past the end ("address of object past end of allocation",
// H5O_protect). open_swmr does not copy that half-way rule for files
// without the SWMR-write flag: they read as File::open reads them.
let script = format!(
r#"
import h5py
for swmr in (False, True):
try:
with h5py.File("{p}", "r", swmr=swmr, locking=False) as f:
f["a"][()]
except Exception as e:
print("refused", swmr)
else:
print("read", swmr)
"#,
p = path.display()
);
let out = Command::new(python())
.args(["-c", &script])
.output()
.unwrap();
let stdout = String::from_utf8_lossy(&out.stdout);
assert_eq!(
stdout.split_whitespace().collect::<Vec<_>>(),
["refused", "False", "read", "True"],
"{}",
String::from_utf8_lossy(&out.stderr)
);
}
#[test]
fn open_swmr_returns_a_permanent_error_at_once() {
// Not HDF5 at all: SignatureNotFound is not something a writer
// causes, so it is not retried (100 attempts would pause about 0.9 s).
let started = std::time::Instant::now();
let err = File::open_storage_swmr(Arc::new(vec![7u8; 4096])).unwrap_err();
assert!(
matches!(
err,
clawhdf5::Error::Format(clawhdf5_format::error::FormatError::SignatureNotFound)
),
"{err}"
);
assert!(started.elapsed() < std::time::Duration::from_millis(100));
// A live file: a lookup of a name it does not have fails at once, and
// a torn object header read (below) is retried.
let f = File::open_storage_swmr(Arc::new(mid_write_copy_with_flags(0x05))).unwrap();
assert!(f.is_swmr_read());
let started = std::time::Instant::now();
assert!(f.dataset("no_such").is_err());
assert!(started.elapsed() < std::time::Duration::from_millis(100));
assert_eq!(f.swmr_retries(), 0);
}
/// A storage whose next `torn` reads come back garbled, as a read racing a
/// rewrite of the structure can see them: the middle byte changed, or with
/// `invert` every byte (signatures included).
struct Torn {
bytes: Vec<u8>,
torn: AtomicU32,
invert: AtomicBool,
}
impl clawhdf5::Storage for Torn {
fn read_at(
&self,
offset: u64,
len: usize,
) -> Result<std::borrow::Cow<'_, [u8]>, clawhdf5_format::error::FormatError> {
let got = self.bytes.as_slice().read_at(offset, len)?;
let tear = self
.torn
.fetch_update(Ordering::SeqCst, Ordering::SeqCst, |n| n.checked_sub(1))
.is_ok();
if tear && !got.is_empty() {
let mut v = got.into_owned();
if self.invert.load(Ordering::SeqCst) {
v.iter_mut().for_each(|b| *b = !*b);
} else {
let mid = v.len() / 2;
v[mid] ^= 0x5a;
}
return Ok(v.into());
}
Ok(got)
}
fn len(&self) -> u64 {
self.bytes.len() as u64
}
}
#[test]
fn a_torn_metadata_read_is_retried_and_never_returned() {
use clawhdf5::Storage as _;
let torn = Arc::new(Torn {
bytes: std::fs::read(fixture("swmr_mid_write.h5")).unwrap(),
torn: AtomicU32::new(0),
invert: AtomicBool::new(false),
});
let f = File::open_storage_swmr(torn.clone()).unwrap();
assert_eq!(f.swmr_retries(), 0);
assert!(torn.len() > 0);
// The next 3 reads (object headers) come back garbled: the lookup
// fails its checks, is run again, and returns the right dataset. The
// same with every byte garbled, signatures included.
for invert in [false, true] {
torn.invert.store(invert, Ordering::SeqCst);
let before = f.swmr_retries();
torn.torn.store(3, Ordering::SeqCst);
let a = f.dataset("a").unwrap();
assert!(f.swmr_retries() > before, "invert {invert}");
assert_eq!(a.read_i64().unwrap(), (1..=148).collect::<Vec<i64>>());
// Two garbled reads: the prefix of the root group's header (whose
// garbled byte may go unused, as it is read again whole) and the
// whole header, which fails its checksum.
let before = f.swmr_retries();
torn.torn.store(2, Ordering::SeqCst);
let b = f.dataset("b").unwrap();
assert_eq!(b.read_f64().unwrap().len(), 148 * 4);
assert!(f.swmr_retries() > before, "invert {invert}");
}
torn.invert.store(false, Ordering::SeqCst);
// With one attempt the error is returned instead (both reads of the
// header garbled, as above).
let mut f = File::open_storage_swmr(torn.clone()).unwrap();
f.set_swmr_read_attempts(1);
torn.torn.store(2, Ordering::SeqCst);
assert!(f.dataset("a").is_err());
assert_eq!(f.swmr_retries(), 0);
torn.torn.store(0, Ordering::SeqCst);
// A refresh that keeps failing gives up after the attempts, and the
// handle keeps its extent.
f.set_swmr_read_attempts(5);
let mut b = f.dataset("b").unwrap();
torn.torn.store(1000, Ordering::SeqCst);
assert!(b.refresh().is_err());
assert_eq!(f.swmr_retries(), 4);
torn.torn.store(0, Ordering::SeqCst);
assert_eq!(b.shape().unwrap(), vec![148, 4]);
b.refresh().unwrap();
assert_eq!(b.read_f64().unwrap().len(), 148 * 4);
// A file not opened for SWMR reading does not retry.
let plain = File::open_storage(torn.clone()).unwrap();
torn.torn.store(2, Ordering::SeqCst);
assert!(plain.dataset("a").is_err());
assert_eq!(plain.swmr_retries(), 0);
}
// ---------------------------------------------------------------------------
// A live file: an h5py SWMR writer appends while clawhdf5 and h5py read.
// ---------------------------------------------------------------------------
/// The h5py SWMR writer. It creates three chunked datasets whose values are
/// a function of their position, so a reader can check every value it
/// reads without knowing when it was written:
///
/// - `a`: int64 `(n,)`, chunks of 100, no filter — `a[i] = i + 1`
/// (Extensible Array index);
/// - `b`: float64 `(n, 4)`, chunks of 16 x 4, gzip — `b[i, j] = 10 i + j + 1`
/// (Extensible Array index);
/// - `c`: int32 `(r, k)`, both unlimited, chunks of 8 x 8, no filter —
/// `c[i, j] = 1000 i + j + 1` (version-2 B-tree index).
///
/// It switches to SWMR mode, prints `ready`, then for `steps` steps appends a
/// random number of rows to each dataset (and every 7th step a column to
/// `c`), resizing before writing as SWMR writers must, flushing each
/// dataset and pausing 2 ms, and finally closes the file and writes the
/// final contents to `<path>.<name>.bin`. `CLAWHDF5_SWMR_STEPS` sets the
/// number of steps (default 2500).
const WRITER: &str = r#"
import sys, time, random, h5py, numpy as np
path, steps = sys.argv[1], int(sys.argv[2])
rng = random.Random(1234)
f = h5py.File(path, "w", libver="latest")
a = f.create_dataset("a", shape=(0,), maxshape=(None,), chunks=(100,), dtype="i8")
b = f.create_dataset("b", shape=(0, 4), maxshape=(None, 4), chunks=(16, 4), dtype="f8",
compression="gzip")
c = f.create_dataset("c", shape=(0, 1), maxshape=(None, None), chunks=(8, 8), dtype="i4")
f.swmr_mode = True
print("ready", flush=True)
na = nb = rc = 0
kc = 1
for step in range(steps):
k = rng.randint(1, 60)
a.resize((na + k,)); a[na:na + k] = np.arange(na, na + k) + 1; na += k
a.flush()
k = rng.randint(1, 20)
rows = np.arange(nb, nb + k)[:, None] * 10 + np.arange(4) + 1
b.resize((nb + k, 4)); b[nb:nb + k] = rows; nb += k
b.flush()
if step % 7 == 6:
c.resize((rc, kc + 1))
if rc:
c[:, kc] = np.arange(rc) * 1000 + kc + 1
kc += 1
k = rng.randint(1, 3)
c.resize((rc + k, kc))
c[rc:rc + k] = np.arange(rc, rc + k)[:, None] * 1000 + np.arange(kc) + 1
rc += k
c.flush()
time.sleep(0.002)
f.close()
with h5py.File(path, "r") as f:
for name in "abc":
open(f"{path}.{name}.bin", "wb").write(f[name][()].tobytes())
print(name, *f[name].shape, flush=True)
"#;
/// h5py's own SWMR reader, run beside ours as the reference: the same
/// checks, until `<path>.stop` exists. Prints its iteration count.
const H5PY_READER: &str = r#"
import os, sys, h5py, numpy as np
path = sys.argv[1]
f = h5py.File(path, "r", swmr=True)
ds = {n: f[n] for n in "abc"}
last = {n: (0,) * ds[n].ndim for n in "abc"}
its = 0
while True:
stop = os.path.exists(path + ".stop")
for n, d in ds.items():
d.refresh()
shape = d.shape
assert all(s >= l for s, l in zip(shape, last[n])), (n, shape, last[n])
last[n] = shape
v = d[()]
if n == "a":
want = np.arange(shape[0]) + 1
elif n == "b":
want = np.arange(shape[0])[:, None] * 10 + np.arange(4) + 1
else:
want = np.arange(shape[0])[:, None] * 1000 + np.arange(shape[1]) + 1
bad = np.argwhere(v != want)
assert len(bad) == 0, (n, shape, bad[:5], v[tuple(bad[0])], want[tuple(bad[0])])
its += 1
if stop:
break
print("iterations", its, *last["a"], *last["b"], *last["c"], flush=True)
"#;
fn want_a(n: u64) -> Vec<i64> {
(1..=n as i64).collect()
}
fn want_b(rows: std::ops::Range<u64>) -> Vec<f64> {
rows.flat_map(|i| (0..4).map(move |j| (10 * i + j + 1) as f64))
.collect()
}
fn want_c(rows: std::ops::Range<u64>, cols: u64) -> Vec<i32> {
rows.flat_map(|i| (0..cols).map(move |j| (1000 * i + j + 1) as i32))
.collect()
}
/// One pass of the Rust reader: refresh every dataset, check its extent
/// did not shrink, and check what it reads (the last rows of each, and all
/// of them every `full_every` passes).
fn check_pass(
datasets: &mut [clawhdf5::Dataset<'_>; 3],
last: &mut [Vec<u64>; 3],
pass: u64,
full_every: u64,
) {
for (k, d) in datasets.iter_mut().enumerate() {
d.refresh().unwrap_or_else(|e| panic!("refresh {k}: {e}"));
let shape = d.shape().unwrap();
assert!(
shape.iter().zip(&last[k]).all(|(s, l)| s >= l),
"dataset {k} shrank: {shape:?} after {:?}",
last[k]
);
last[k] = shape;
}
let full = pass.is_multiple_of(full_every);
let na = last[0][0];
let from = if full { 0 } else { na.saturating_sub(200) };
let got = if full {
datasets[0].read_i64()
} else {
datasets[0].read_i64_selection(&Selection::slice(std::slice::from_ref(&(from..na))))
}
.unwrap_or_else(|e| panic!("read a: {e}"));
let want: Vec<i64> = want_a(na).split_off(from as usize);
assert!(got == want, "a {from}..{na}: {:?}", first_diff(&got, &want));
let nb = last[1][0];
let from = if full { 0 } else { nb.saturating_sub(50) };
let sel = Selection::slice(&[from..nb, 0..4]);
let got = datasets[1]
.read_f64_selection(&sel)
.unwrap_or_else(|e| panic!("read b: {e}"));
let want = want_b(from..nb);
assert!(got == want, "b {from}..{nb}: {:?}", first_diff(&got, &want));
let (rc, kc) = (last[2][0], last[2][1]);
let from = if full { 0 } else { rc.saturating_sub(9) };
let got = if full {
datasets[2].read_i32()
} else {
datasets[2].read_i32_selection(&Selection::slice(&[from..rc, 0..kc]))
}
.unwrap_or_else(|e| panic!("read c: {e}"));
let want = want_c(from..rc, kc);
assert!(
got == want,
"c {from}..{rc} x {kc}: {:?}",
first_diff(&got, &want)
);
}
fn first_diff<T: PartialEq + std::fmt::Debug>(got: &[T], want: &[T]) -> String {
if got.len() != want.len() {
return format!("{} values, want {}", got.len(), want.len());
}
let i = got.iter().zip(want).position(|(g, w)| g != w).unwrap();
format!("first difference at {i}: {:?}, want {:?}", got[i], want[i])
}
#[test]
fn a_live_file_reads_consistently_while_an_h5py_swmr_writer_appends() {
skip_if_no_python!();
let steps: u32 = std::env::var("CLAWHDF5_SWMR_STEPS")
.ok()
.and_then(|s| s.parse().ok())
.unwrap_or(2500);
let dir = tempfile::tempdir_in(env!("CARGO_TARGET_TMPDIR")).unwrap();
let path = dir.path().join("live.h5");
let mut writer = Command::new(python())
.args(["-c", WRITER, path.to_str().unwrap(), &steps.to_string()])
.stdout(Stdio::piped())
.spawn()
.unwrap();
let mut out = BufReader::new(writer.stdout.take().unwrap());
let mut line = String::new();
out.read_line(&mut line).unwrap();
assert_eq!(line.trim(), "ready", "writer did not start");
// h5py's SWMR reader on the same file, as the reference.
let h5py_reader = Command::new(python())
.args(["-c", H5PY_READER, path.to_str().unwrap()])
.stdout(Stdio::piped())
.stderr(Stdio::piped())
.spawn()
.unwrap();
let file = File::open_swmr(&path).unwrap();
assert!(file.is_swmr_read());
assert!(file.swmr_writer_active().unwrap());
// Two reader threads share the file, each with its own handles; each
// makes passes until it has seen the writer exit, then one more.
let writer_done = AtomicBool::new(false);
let reader = || {
let mut datasets = ["a", "b", "c"].map(|n| file.dataset(n).unwrap());
let mut last = [vec![0], vec![0, 4], vec![0, 1]];
let (mut passes, mut grew) = (0u64, 0u64);
loop {
let done = writer_done.load(Ordering::Acquire);
let before = last[0][0];
check_pass(&mut datasets, &mut last, passes, 50);
passes += 1;
grew += u64::from(last[0][0] > before);
if done {
break;
}
}
// The writer has closed the file: one more pass reads the final
// extents, all of them.
check_pass(&mut datasets, &mut last, 0, 50);
(passes, grew, last)
};
let (status, results) = std::thread::scope(|s| {
let readers = [s.spawn(reader), s.spawn(reader)];
let status = writer.wait().unwrap();
writer_done.store(true, Ordering::Release);
(status, readers.map(|r| r.join().unwrap()))
});
assert!(status.success(), "writer failed");
assert!(!file.swmr_writer_active().unwrap());
let last = results[0].2.clone();
assert_eq!(results[1].2, last);
let passes: Vec<u64> = results.iter().map(|r| r.0).collect();
let grew: u64 = results.iter().map(|r| r.1).min().unwrap();
std::fs::write(path.with_extension("h5.stop"), b"").unwrap();
let reference = h5py_reader.wait_with_output().unwrap();
assert!(
reference.status.success(),
"h5py's SWMR reader failed:\n{}",
String::from_utf8_lossy(&reference.stderr)
);
let reference = String::from_utf8_lossy(&reference.stdout).into_owned();
// The writer's final contents, as h5py reads the closed file.
let mut finals = String::new();
out.read_to_string(&mut finals).unwrap();
let shapes: std::collections::HashMap<&str, Vec<u64>> = finals
.lines()
.map(|l| {
let mut w = l.split_whitespace();
let name = w.next().unwrap();
(name, w.map(|v| v.parse().unwrap()).collect())
})
.collect();
assert_eq!(shapes["a"], last[0]);
assert_eq!(shapes["b"], last[1]);
assert_eq!(shapes["c"], last[2]);
let bin = |name: &str| std::fs::read(format!("{}.{name}.bin", path.display())).unwrap();
let le = |b: &[u8], n: usize| -> Vec<[u8; 8]> {
b.chunks(n)
.map(|c| {
let mut v = [0u8; 8];
v[..n].copy_from_slice(c);
v
})
.collect()
};
// The live handle and a fresh non-SWMR open of the closed file both
// read exactly what h5py reads.
for f in [&file, &File::open(&path).unwrap()] {
let a: Vec<[u8; 8]> = f
.dataset("a")
.unwrap()
.read_i64()
.unwrap()
.iter()
.map(|v| v.to_le_bytes())
.collect();
assert_eq!(a, le(&bin("a"), 8));
let b: Vec<[u8; 8]> = f
.dataset("b")
.unwrap()
.read_f64()
.unwrap()
.iter()
.map(|v| v.to_le_bytes())
.collect();
assert_eq!(b, le(&bin("b"), 8));
let c: Vec<[u8; 8]> = f
.dataset("c")
.unwrap()
.read_i32()
.unwrap()
.iter()
.map(|v| {
let mut x = [0u8; 8];
x[..4].copy_from_slice(&v.to_le_bytes());
x
})
.collect();
assert_eq!(c, le(&bin("c"), 4));
}
eprintln!(
"SWMR: {steps} writer steps; clawhdf5 readers made {passes:?} passes (each saw `a` \
grow at least {grew} times), {} retries; h5py reader: {}",
file.swmr_retries(),
reference.trim()
);
assert!(
grew >= 2,
"a reader never saw the file grow ({passes:?} passes)"
);
}
// ---------------------------------------------------------------------------
// Every read path retries: strings, variable-length data, attributes.
// ---------------------------------------------------------------------------
/// A storage over `bytes` whose read number `fail_at` (counted from the
/// last [`Flaky::arm`]) fails with a checksum mismatch, as a read racing a
/// SWMR writer's flush can; it counts the reads.
struct Flaky {
bytes: Vec<u8>,
reads: AtomicU32,
fail_at: AtomicU32,
}
impl Flaky {
fn arm(&self, fail_at: u32) {
self.reads.store(0, Ordering::SeqCst);
self.fail_at.store(fail_at, Ordering::SeqCst);
}
}
impl clawhdf5::Storage for Flaky {
fn read_at(
&self,
offset: u64,
len: usize,
) -> Result<std::borrow::Cow<'_, [u8]>, clawhdf5_format::error::FormatError> {
let n = self.reads.fetch_add(1, Ordering::SeqCst);
if n == self.fail_at.load(Ordering::SeqCst) {
return Err(clawhdf5_format::error::FormatError::ChecksumMismatch {
expected: 0,
computed: 1,
});
}
self.bytes.as_slice().read_at(offset, len)
}
fn len(&self) -> u64 {
self.bytes.len() as u64
}
}
/// `swmr_strings_attrs.h5`: a copy h5py 3.16 (HDF5 2.0) made of its own
/// file right after switching it to SWMR mode (the SWMR-write flag is set),
/// holding a variable-length string dataset `s` (`alpha beta gamma
/// delta`), a variable-length int32 dataset `v` (`[1] [2 3] [4 5 6]`), an
/// int64 dataset `d` (`0..10`) with 12 string attributes `k00`..`k11`
/// (`value 0`..; dense storage: a fractal heap and a v2 B-tree), a group
/// `g` with attribute `note`, and root attributes `title` (a
/// variable-length string) and `n`.
#[test]
fn every_read_path_of_a_live_file_retries_a_failed_read() {
let flaky = Arc::new(Flaky {
bytes: std::fs::read(fixture("swmr_strings_attrs.h5")).unwrap(),
reads: AtomicU32::new(0),
fail_at: AtomicU32::new(u32::MAX),
});
let f = File::open_storage_swmr(flaky.clone()).unwrap();
assert!(f.is_swmr_read());
let root = f.root();
let g = f.group("g").unwrap();
let s = f.dataset("s").unwrap();
let v = f.dataset("v").unwrap();
let d = f.dataset("d").unwrap();
let sorted = |m: std::collections::HashMap<String, clawhdf5::AttrValue>| {
format!(
"{:?}",
m.into_iter().collect::<std::collections::BTreeMap<_, _>>()
)
};
type Op<'a> = Box<dyn Fn() -> Result<String, clawhdf5::Error> + 'a>;
let ops: Vec<(&str, Op<'_>)> = vec![
("root.attrs", Box::new(|| root.attrs().map(sorted))),
(
"root.attr",
Box::new(|| root.attr("title").map(|a| format!("{a:?}"))),
),
("g.attrs", Box::new(|| g.attrs().map(sorted))),
("d.attrs", Box::new(|| d.attrs().map(sorted))),
(
"d.attr",
Box::new(|| d.attr("k05").map(|a| format!("{a:?}"))),
),
(
"root.datasets",
Box::new(|| root.datasets().map(|n| format!("{n:?}"))),
),
(
"root.groups",
Box::new(|| root.groups().map(|n| format!("{n:?}"))),
),
("d.shape", Box::new(|| d.shape().map(|n| format!("{n:?}")))),
(
"d.read_i64",
Box::new(|| d.read_i64().map(|n| format!("{n:?}"))),
),
(
"s.read_string",
Box::new(|| s.read_string().map(|n| format!("{n:?}"))),
),
(
"s.read_string_bytes",
Box::new(|| s.read_string_bytes().map(|n| format!("{n:?}"))),
),
(
"s.read_string_selection",
Box::new(|| {
s.read_string_selection(&Selection::slice(std::slice::from_ref(&(1..3))))
.map(|n| format!("{n:?}"))
}),
),
(
"v.read_vlen",
Box::new(|| v.read_vlen::<i32>().map(|n| format!("{n:?}"))),
),
(
"v.read_vlen_selection",
Box::new(|| {
v.read_vlen_selection::<i32>(&Selection::slice(std::slice::from_ref(&(1..3))))
.map(|n| format!("{n:?}"))
}),
),
];
for (name, op) in &ops {
flaky.arm(u32::MAX);
let want = op().unwrap_or_else(|e| panic!("{name}: {e}"));
let reads = flaky.reads.load(Ordering::SeqCst);
// Each read the operation makes fails once in turn: the operation
// is run again and returns the same result.
for k in 0..reads {
flaky.arm(k);
let before = f.swmr_retries();
let got = op().unwrap_or_else(|e| panic!("{name}, read {k} of {reads} failing: {e}"));
assert_eq!(got, want, "{name}, read {k} of {reads} failing");
assert_eq!(f.swmr_retries(), before + 1, "{name}, read {k} of {reads}");
}
}
flaky.arm(u32::MAX);
assert!(ops[0].1().unwrap().contains("live strings"));
assert!(ops[3].1().unwrap().contains("value 11"));
assert_eq!(
ops[9].1().unwrap(),
r#"["alpha", "beta", "gamma", "delta"]"#
);
assert_eq!(ops[12].1().unwrap(), "[[1], [2, 3], [4, 5, 6]]");
// With one attempt the failure reaches the caller.
let mut f = File::open_storage_swmr(flaky.clone()).unwrap();
f.set_swmr_read_attempts(1);
let s = f.dataset("s").unwrap();
flaky.arm(0);
assert!(s.read_string().is_err());
}
docs/design/range-reads.md
+16 -1
View File
@@ -7,7 +7,9 @@ change. Progress: M0 and M1 are done, and so is M2 (branch
`File::open_storage` gives the facade's read API over any `Storage` (see
`CHANGELOG.md`, "Range reads, milestone M2"). M3 is done on branch
`feat/p3-m3-remote`: the `clawhdf5-remote` crate (block cache, HTTP(S),
object stores) and URLs in `h5rs` (see the M3 status below). M4 (wasm) is
object stores) and URLs in `h5rs` (see the M3 status below). M5 (SWMR) is
done on branch `feat/p3-m5-swmr-reader`, with its own design in
[`swmr.md`](swmr.md) (see the M5 status below). M4 (wasm) is
next. Every count in §1–§2 was
change. Progress: M1, first part (the `Storage` trait and the metadata
@@ -523,6 +525,19 @@ fast path within benchmark noise.
**M5 — SWMR and growth (later, separate design).** `Storage::len()` may grow;
add `File::refresh()` that re-reads the superblock/EOF and invalidates cached
blocks past the old end. Needs libhdf5 SWMR semantics research first.
- *Status 2026-09-27:* done on branch `feat/p3-m5-swmr-reader`; design and
libhdf5 research in [`swmr.md`](swmr.md). Differences from the sketch
above: the refresh is per dataset (`Dataset::refresh`, as libhdf5's
`H5Drefresh`), not per file — a SWMR writer only grows datasets, and the
superblock's EOF is not kept up to date by it, so there is nothing to
re-read there (`File::swmr_writer_active` re-reads its flags). A live
file (`File::open_swmr`) reads through `FileStorage` (positioned reads,
`len()` the current length) with no block or chunk cache, rather than
invalidating cached blocks: `BlockCache`/`HttpStorage` stay snapshot
readers. Operations that fail with an error a racing write can cause are
retried up to 100 times (libhdf5's default for SWMR readers).
Tested against a live h5py writer and h5py's SWMR reader
(`crates/clawhdf5/tests/swmr_interop.rs`).
Total: roughly 6–10 engineer-weeks for M0–M4 (estimate, not measured).
docs/design/swmr.md
+178
View File
@@ -0,0 +1,178 @@
# Design: reading files a SWMR writer is still appending to (range-read M5)
Status: design 2026-09-27, implemented on branch `feat/p3-m5-swmr-reader`
(see "Status" at the end). This is milestone M5 of
[`range-reads.md`](range-reads.md): "`Storage::len()` may grow; add a
refresh". It covers the reader only; clawhdf5 does not write SWMR files.
## What libhdf5 does
A SWMR ("single writer, multiple readers") writer is a libhdf5 process that
opened a file with `libver='latest'` and switched to SWMR mode (h5py
`f.swmr_mode = True`, `H5Fstart_swmr_write`). Readers open the same file
with `H5F_ACC_SWMR_READ` (h5py `File(path, 'r', swmr=True)`) while the
writer keeps appending. What the format and the library guarantee:
- **Superblock v3, flags set.** The writer sets the superblock's
file-consistency flags to write access + SWMR write (`0x05`) and clears
them on close. The superblock's end-of-file address is *not* kept up to
date while writing: a copy of a file taken mid-write records an EOF of a
few hundred bytes while the file is tens of kilobytes (checked on tank,
2026-09-27, h5py 3.16 / HDF5 2.0: EOF 715 in a 17 857-byte file). A SWMR
reader therefore skips libhdf5's end-of-allocation check for every read
(`H5FD_read`: "allow access to data past the end of the allocated space
… for SWMR read access"), and bounds reads by the file's real length.
- **A non-SWMR open of such a file fails** in libhdf5: "file is already
open for write (may use <h5clear file> to clear file consistency flags)".
- **The writer only appends.** New objects and attributes cannot be created
in SWMR mode; datasets grow with `H5Dset_extent` and are written. Chunked
datasets with one unlimited dimension use an Extensible Array index, with
more than one a version-2 B-tree; both are updated in a SWMR-safe way.
Fixed Array and single-chunk indexes are for datasets that cannot grow.
- **Flush ordering.** Every metadata structure the writer uses is
checksummed, and flush dependencies order the writes: a chunk's data is
written before the index entry that points to it, and index blocks before
the object header whose dataspace announces the new extent. A reader
that reads the object header first and the index after sees an index at
least as new as the extent, so every chunk inside the extent it read is
either in the index or never written (then it reads as the fill value, as
it does for libhdf5's reader).
- **Refresh.** A reader sees a dataset's new extent only when it refreshes
it (`H5Drefresh`, h5py `Dataset.refresh()`), which evicts the dataset's
cached metadata and reads the object header again.
- **Retries.** Reads are not atomic against writes on every system, so a
checksum can fail when a structure is read while the writer rewrites it.
A SWMR reader reads checksummed metadata up to 100 times before failing
(`H5Pset_metadata_read_attempts`; default 100 for SWMR access, 1
otherwise — `H5Ppublic.h` of HDF5 1.14.6).
## What clawhdf5 did before
- `File::open` of a SWMR-flagged file bounded every read by the recorded
EOF (`Superblock::data_end` only tolerated an EOF *past* the end of the
file). A file copied or read mid-write therefore listed, but every
chunked read failed ("unexpected EOF: need 787 bytes, have 715"), and
`h5rs check` reported chunk indexes "past the end of the file".
- A `File` is a snapshot: an mmap (or a buffer) of the length at open, and a
per-file chunk cache that keeps each dataset's chunk index and decoded
chunks for the life of the `File`. A reader could not see growth at all,
and a mapping of a file that is being rewritten can change under a read.
## Design
1. **Bound SWMR files by their length.** `Superblock::data_end` returns the
file's length for a version-3 superblock with the SWMR-write flag,
whatever EOF it records, as libhdf5's SWMR reader does. Every existing
open path (`File::open`, `open_storage`, `h5rs`) then reads a finished
copy of a live file. We keep opening such files without a SWMR flag
(libhdf5 refuses): it is read-only and the alternative is an error.
2. **A live open: `File::open_swmr(path)` / `File::open_storage_swmr`.**
- The file is read with positioned reads (`FileStorage`, `pread` on
Unix, `seek_read` on Windows), never mapped, and `len()` is the file's
current length, so reads past the length seen at open work.
- The facade's view of the file (`FileData`) is *live*: reads are not
clamped to an end fixed at open, only by the storage's current length.
- The chunk cache is not used: every read reads the chunk index and the
chunks it needs again. A cached index would hide new chunks, and a
cached partial edge chunk would read as fill where the writer has since
written data (unfiltered edge chunks are rewritten in place).
- Only a file whose superblock has the SWMR-write flag when it is opened
is read live. Any other file is read exactly as `File::open` reads it
(bounded by its recorded EOF, through the chunk cache, no retries), and
`is_swmr_read()` is `false`. libhdf5's SWMR reader is looser: it skips
the end-of-allocation check in `H5FD_read` for every file it opens,
flagged or not, yet still refuses an object header past the EOF
(`H5O_protect`, "address of object past end of allocation"). On a
closed file whose EOF is below its length (checked 2026-09-27, h5py
3.16 / HDF5 2.0, the mid-write fixture with its flags cleared) it
therefore reads a dataset whose chunk index lies past the EOF, which
its plain reader refuses. We follow the plain reader there: the EOF of
a file no SWMR writer has open is what the file says it is.
- A storage that caches blocks (`clawhdf5-remote`'s `BlockCache`) would
serve stale bytes; `HttpStorage` also pins a file by ETag and length and
refuses a changed file. Remote SWMR is out of scope.
3. **`Dataset::refresh()`** reads the dataset's object header again (same
address) and replaces the handle's copy, so `shape()` and every later
read use the new extent. Like h5py, a handle that is not refreshed keeps
its extent; its reads still read the index as it is now, and only return
elements inside that extent.
4. **Bounded retries.** In a live file, an operation (open, lookups and
listings, refresh, every dataset read — typed, raw, selections, strings
and variable-length data with their global-heap decoding — attributes,
`File::decode_*`, `verify_provenance`) that fails with an error a
concurrent write can
cause is run again from the start, up to `File::swmr_read_attempts()`
times (default 100, libhdf5's default; `set_swmr_read_attempts` changes
it), sleeping 1 µs, 2 µs, … up to 10 ms between attempts (under a second
in all). libhdf5 retries the one structure whose checksum failed; we
retry the whole operation, because the parsers are pure functions of the
bytes they read. Which errors: those libhdf5 retries (`H5C__load_entry`)
— a checksum mismatch, and a failure to decode the prefix it reads
before the checksum to size the structure (for an object header its
signature and version: a test that garbles every byte of a header gets
`InvalidObjectHeaderVersion`) — and a read past the file's current end
(short here; libhdf5 reads zeros there, which fail the checksum). Every
other error (an unsupported version or message, a file that is not HDF5,
a structure corrupt behind a valid checksum) is returned at once: an
earlier version retried nearly every format error, so a permanent one
cost about 0.9 s of pauses. Non-live files never retry.
Results are only returned from a run where every structure verified, so
a torn metadata read is an error, never data. `File::swmr_retries()`
counts the retries (libhdf5: `H5Fget_metadata_read_retry_info`).
Attribute reads leave out an attribute they cannot read (or return a
variable-length string one as `AttrValue::Raw`) instead of failing;
on a live file such an error of a retried kind runs the read again too,
and after the last attempt the last result is returned as before. The
zero-copy reads (`read_raw_ref`, `read_as_slice`, `read_*_zerocopy`)
need the file in memory, which a live file never is: they report
`None` / `ContiguousStorageRequired` without reading data.
Global heap collections (variable-length data) have no checksum in
HDF5, like raw data, so a torn read of one is only caught when it fails
a check (its signature, a bound).
Raw data has no checksum in HDF5 (unless Fletcher-32 is on), in libhdf5
as here: correctness rests on the writer's ordering (chunk data before
the index entry, only appends), as for libhdf5's reader.
5. **Writer state.** `File::swmr_writer_active()` reads the superblock
flags again, so a reader can tell when the writer has closed the file.
A writer that crashed or was killed never clears the flag, so a reader
that follows a file needs a second stop condition (the README example
stops after a minute without growth).
What stays out: SWMR writing, VFD SWMR (HDF5 1.13's page-buffer protocol,
not in 1.14 or 2.0), remote SWMR, refresh of groups/attributes (a SWMR
writer cannot add them), and `MmapFile`/`LazyFile`.
## Tests
- `crates/clawhdf5-format`: `data_end` of a SWMR-flagged v3 superblock whose
EOF is below the file length.
- `crates/clawhdf5/tests/swmr_interop.rs`:
- a copy of a file taken mid-write (fixture) reads like h5py's SWMR reader;
- the same bytes with the flags cleared read as `File::open` reads them
(and how h5py's two readers read them);
- a permanent error returns at once; every read path (listings,
attributes, strings, variable-length data) with each of its reads
failing once in turn returns the same result;
- a live test: an h5py writer (`swmr_mode = True`) appends to a 1-D and a
2-D dataset with one unlimited dimension (Extensible Array, one of them
gzip) and a 2-D dataset with two (v2 B-tree), flushing after every step,
while a Rust reader refreshes and reads them in a loop, and an h5py SWMR
reader does the same as the reference. Every value read must be the value
the writer wrote (a deterministic function of its position), extents
never shrink, and after the writer closes both readers must read the
same data as h5py.
## Status
Implemented 2026-09-27 on branch `feat/p3-m5-swmr-reader` as designed
above (`CHANGELOG.md`, "Range reads, milestone M5"). Observed on tank the
same day (h5py 3.16 / HDF5 2.0, `cargo test -p clawhdf5 --test
swmr_interop`, and once with `CLAWHDF5_SWMR_STEPS=20000` in a release
build): no read returned a value the writer had not written at that
position, h5py's reader agreed, and retries were needed but rare (the
failures seen were checksum mismatches, each cured by one retry). A
variant of the test with the chunk cache left on in live mode fails it
(stale chunk index / edge chunk), which is why live files do not use it.
Also found: `File::open` of such a file had been failing since the
end-of-file check of 2026-09-26 (item 1; `docs/known-issues.md`).
docs/known-issues.md
+23
View File
@@ -7,6 +7,29 @@ deleting it.
---
## Files a SWMR writer had open could not be read past a stale end of file
**Status:** fixed 2026-09-27 (branch `feat/p3-m5-swmr-reader`), before any
release: reads have been bounded by the recorded end of file since
`7d7a7e7` (2026-09-26), which no release contains.
A libhdf5 writer in SWMR mode (h5py `f.swmr_mode = True`) sets the
superblock's SWMR-write flag and does not keep its end-of-file address up to
date: a copy h5py made of its own file mid-write records 715 in a
6 030-byte file (h5py 3.16 / HDF5 2.0, tank). `Superblock::data_end` only
ignored the recorded end when it lay *past* the end of the file, so every
reader bounded such a file at 715 bytes: it listed, but every chunked read
failed ("unexpected EOF: need 787 bytes, have 715") and `h5rs check`
reported the chunk indexes past the end of the file. Never wrong data.
**Fix:** for a v3 superblock with the SWMR-write flag, the data ends at the
end of the file, as libhdf5's SWMR reader reads it. **Test:**
`crates/clawhdf5/tests/swmr_interop.rs` (the mid-write copy,
`tests/fixtures/swmr_mid_write.h5`, through every open path and against
h5py's SWMR reader). A file still being written is read with
`File::open_swmr` (see `docs/design/swmr.md`); `File::open` maps the file
at its length at open and is not meant for files that change while open.
## Fletcher-32 checksums disagreed with libhdf5 on about 1 chunk in 32768
**Status:** fixed 2026-09-26, after v2.7.0. **Every release (v2.1.0 to