quantumclaw/clawhdf5
3
1
Fork 0
Code Issues Pull Requests 3 Actions Packages Projects Releases 1 Wiki Activity

py: in-place editing (clawhdf5.File(path, 'r+')) through FileEditor

Browse Source 
clawhdf5.File(path, 'r+') (and 'a' on an existing file) holds a
FileEditor, and with it the file's exclusive lock, until close():

- ds[key] = value: h5py's keys and broadcasting (numpy's rules for
  slices and integers with extra leading 1-axes allowed; the exact shape
  for an index list, a scalar only where h5py expands it). Arrays are
  converted as libhdf5 converts them in native byte order (integers
  saturate, floats truncate toward zero and clip, integers go into h5py's
  bool enum by value); other values through
  numpy.asarray(value, dtype=ds.dtype), as h5py does. NaN into an integer
  dataset is a ValueError instead of libhdf5's arbitrary value. The value
  preparation is a small Python module compiled into the extension
  (src/edit_helpers.py).
- ds.resize(shape) / ds.resize(n, axis=k) with h5py's argument rules.
- attrs[name] = value, attrs.create(name, data, shape, dtype),
  attrs.modify: numeric, bool, complex, bytes and str data of any shape,
  with h5py's HDF5 types; str is stored as fixed-length UTF-8 (the editor
  cannot write variable-length strings).
- File.mode, File.flush(), Dataset.chunks.

Each edit runs with the GIL released under the file handle's write lock
(no read sees a half-written edit), then the file is reopened;
datasets and attrs objects re-read their shape and attributes when the
handle's edit generation moved. What the editor cannot do is
NotImplementedError before anything is written: deleting attributes or
objects, creating datasets or groups, compound fields by name,
variable-length data, and FileEditor's own limits.

Where libhdf5 2.0 (h5py 3.16) converts inconsistently -- its soft
conversions in non-native byte order (a float in (-1, 0) becomes the
integer minimum, same-size unsigned->signed wraps) and native casts that
are undefined in C (half floats into unsigned, float(max) rounded up) --
clawhdf5 saturates as libhdf5's native path does; listed in
docs/known-issues.md.

Tests (tests/test_edit.py): every edit applied by h5py and by clawhdf5 to
copies of the same file and both read back through h5py after each edit,
on h5py files (libver earliest, v114, latest) and a clawhdf5 file: a fixed
sequence over every chunk index kind, compact/contiguous/gzip layouts and
numeric, bool, enum, complex, string and compound types, 16 random
sequences of 40 edits, and a numeric conversion matrix; a refused edit
must be refused by both and leave the file unchanged. Also dense
attributes, locking, objects seeing edits, readers racing a writer, and
h5dump (plus h5rs check in ci-test.sh) on every edited file. The
read-vs-h5py suite also runs on a file opened 'r+'.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-27 06:57:36 -05:00
co-authored by Claude Opus 5.5
parent 910d81904c
commit 92fb0830e0
15 changed files with 1978 additions and 102 deletions
Download Patch File Download Diff File Expand all files Collapse all files
crates/clawhdf5-py/src/handle.rs
+110 -8
View File
@@ -2,25 +2,34 @@
//!
//! Every read goes through [`Handle::with`], which releases the GIL and
//! parses through `File::storage()`, so the same code serves a local file
//! (memory-mapped) and a remote one (`clawhdf5-remote`: range requests
//! through a block cache, so a network read never holds the GIL).
//! (memory-mapped), a remote one (`clawhdf5-remote`: range requests through
//! a block cache, so a network read never holds the GIL) and a file open
//! for editing.
//!
//! A file opened with `'r+'` also holds a [`FileEditor`]. An edit takes the
//! file's write lock, so no read runs while the file changes underneath it,
//! and reopens the file afterwards: reads after an edit see the new bytes
//! (a grown file, a new dataspace), never a stale mapping or chunk cache.
//! Objects that cache something an edit can change compare
//! [`Handle::generation`] with the value they cached it at.
//!
//! Lock discipline (no deadlock with the GIL): the file lock is only taken
//! with the GIL released, and code that holds it never touches Python.
use std::path::PathBuf;
use std::sync::{Arc, PoisonError, RwLock};
use std::sync::atomic::{AtomicU64, Ordering};
use std::sync::{Arc, Mutex, PoisonError, RwLock};
use clawhdf5_rs::File;
use clawhdf5_rs::{File, FileEditor};
use pyo3::exceptions::PyOSError;
use pyo3::prelude::*;
use crate::to_py_err;
use crate::{panic_text, to_py_err};
/// Where the file's bytes come from.
pub(crate) enum Source {
/// A local path (memory-mapped).
Local(#[allow(dead_code)] PathBuf),
Local(PathBuf),
/// A URL, read through `clawhdf5-remote`'s block cache.
Remote {
url: String,
@@ -29,8 +38,14 @@ pub(crate) enum Source {
}
pub(crate) struct Handle {
/// The file as last opened; `None` if reopening it after an edit failed
/// (every read is then an error rather than a read of stale bytes).
file: RwLock<Option<File>>,
/// For `'r+'`: the editor, until the file is closed.
editor: Option<Mutex<Option<FileEditor>>>,
source: Source,
/// Bumped by every edit.
generation: AtomicU64,
pub offset_size: u8,
pub length_size: u8,
pub root: u64,
@@ -41,13 +56,15 @@ fn closed_after_failed_reopen() -> PyErr {
}
impl Handle {
fn new(file: File, source: Source) -> Arc<Self> {
fn new(file: File, source: Source, editor: Option<FileEditor>) -> Arc<Self> {
let sb = file.superblock();
let (offset_size, length_size, root) =
(sb.offset_size, sb.length_size, sb.root_group_address);
Arc::new(Self {
file: RwLock::new(Some(file)),
editor: editor.map(|e| Mutex::new(Some(e))),
source,
generation: AtomicU64::new(0),
offset_size,
length_size,
root,
@@ -57,7 +74,25 @@ impl Handle {
/// A local file, read-only.
pub(crate) fn open_local(py: Python<'_>, path: &str) -> PyResult<Arc<Self>> {
let file = py.detach(|| crate::no_panic(|| File::open(path).map_err(to_py_err)))?;
Ok(Self::new(file, Source::Local(PathBuf::from(path))))
Ok(Self::new(file, Source::Local(PathBuf::from(path)), None))
}
/// A local file, open for in-place editing (`'r+'`): the editor takes
/// the file's exclusive lock and checks that it can edit the file, then
/// the file is opened for reading.
pub(crate) fn open_editable(py: Python<'_>, path: &str) -> PyResult<Arc<Self>> {
let (file, editor) = py.detach(|| {
crate::no_panic(|| {
let editor = FileEditor::open(path).map_err(to_py_err)?;
let file = File::open(path).map_err(to_py_err)?;
Ok((file, editor))
})
})?;
Ok(Self::new(
file,
Source::Local(PathBuf::from(path)),
Some(editor),
))
}
/// A remote file (`http(s)://`, `s3://`, ...).
@@ -79,6 +114,7 @@ impl Handle {
url: url.to_string(),
storage,
},
None,
))
}
@@ -102,6 +138,17 @@ impl Handle {
})
}
/// Edits so far: objects that cache something an edit can change (a
/// dataset's shape, an object's attributes) re-read it when this moved.
pub(crate) fn generation(&self) -> u64 {
self.generation.load(Ordering::Acquire)
}
/// Whether the file was opened for editing (`'r+'`), even if closed since.
pub(crate) fn is_writable(&self) -> bool {
self.editor.is_some()
}
/// The remote file's block cache.
pub(crate) fn remote_storage(&self) -> Option<&clawhdf5_remote::RemoteStorage> {
match &self.source {
@@ -117,6 +164,61 @@ impl Handle {
Source::Local(_) => None,
}
}
/// Release the editor, and with it the file's lock. Objects still
/// open keep reading the file as it was last written; an edit through
/// them is an error.
pub(crate) fn close(&self) {
if let Some(ed) = &self.editor {
ed.lock().unwrap_or_else(PoisonError::into_inner).take();
}
}
/// Apply one edit with the GIL released. No read runs while it writes,
/// and the file is reopened afterwards — also after a failed edit, since
/// a commit that failed part-way may have changed the file.
pub(crate) fn edit<R: Send>(
&self,
py: Python<'_>,
f: impl FnOnce(&mut FileEditor) -> Result<R, clawhdf5_rs::Error> + Send,
) -> PyResult<R> {
let Some(editor) = &self.editor else {
return Err(PyOSError::new_err(match self.source {
Source::Remote { .. } => "remote files are read-only",
Source::Local(_) => {
"the file is open read-only; open it with mode 'r+' to change it"
}
}));
};
let Source::Local(path) = &self.source else {
return Err(PyOSError::new_err("remote files are read-only"));
};
py.detach(|| {
let mut ed = editor.lock().unwrap_or_else(PoisonError::into_inner);
let ed = ed
.as_mut()
.ok_or_else(|| PyOSError::new_err("the file is closed"))?;
let mut file = self.file.write().unwrap_or_else(PoisonError::into_inner);
let result = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| f(ed)));
// Drop the old mapping and its chunk cache before reopening.
*file = None;
let reopened = std::panic::catch_unwind(|| File::open(path));
self.generation.fetch_add(1, Ordering::AcqRel);
match reopened {
Ok(Ok(f)) => *file = Some(f),
Ok(Err(e)) => return Err(to_py_err(e)),
Err(_) => return Err(closed_after_failed_reopen()),
}
drop(file);
match result {
Ok(r) => r.map_err(to_py_err),
Err(p) => Err(crate::InternalError::new_err(format!(
"clawhdf5 internal error (please report it): {}",
panic_text(&*p)
))),
}
})
}
}
/// A `clawhdf5_remote::Error` as a Python exception: the network side