quantumclaw/clawhdf5
3
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

py: in-place editing (clawhdf5.File(path, 'r+')) through FileEditor

Browse Source 
clawhdf5.File(path, 'r+') (and 'a' on an existing file) holds a
FileEditor, and with it the file's exclusive lock, until close():

- ds[key] = value: h5py's keys and broadcasting (numpy's rules for
  slices and integers with extra leading 1-axes allowed; the exact shape
  for an index list, a scalar only where h5py expands it). Arrays are
  converted as libhdf5 converts them in native byte order (integers
  saturate, floats truncate toward zero and clip, integers go into h5py's
  bool enum by value); other values through
  numpy.asarray(value, dtype=ds.dtype), as h5py does. NaN into an integer
  dataset is a ValueError instead of libhdf5's arbitrary value. The value
  preparation is a small Python module compiled into the extension
  (src/edit_helpers.py).
- ds.resize(shape) / ds.resize(n, axis=k) with h5py's argument rules.
- attrs[name] = value, attrs.create(name, data, shape, dtype),
  attrs.modify: numeric, bool, complex, bytes and str data of any shape,
  with h5py's HDF5 types; str is stored as fixed-length UTF-8 (the editor
  cannot write variable-length strings).
- File.mode, File.flush(), Dataset.chunks.

Each edit runs with the GIL released under the file handle's write lock
(no read sees a half-written edit), then the file is reopened;
datasets and attrs objects re-read their shape and attributes when the
handle's edit generation moved. What the editor cannot do is
NotImplementedError before anything is written: deleting attributes or
objects, creating datasets or groups, compound fields by name,
variable-length data, and FileEditor's own limits.

Where libhdf5 2.0 (h5py 3.16) converts inconsistently -- its soft
conversions in non-native byte order (a float in (-1, 0) becomes the
integer minimum, same-size unsigned->signed wraps) and native casts that
are undefined in C (half floats into unsigned, float(max) rounded up) --
clawhdf5 saturates as libhdf5's native path does; listed in
docs/known-issues.md.

Tests (tests/test_edit.py): every edit applied by h5py and by clawhdf5 to
copies of the same file and both read back through h5py after each edit,
on h5py files (libver earliest, v114, latest) and a clawhdf5 file: a fixed
sequence over every chunk index kind, compact/contiguous/gzip layouts and
numeric, bool, enum, complex, string and compound types, 16 random
sequences of 40 edits, and a numeric conversion matrix; a refused edit
must be refused by both and leave the file unchanged. Also dense
attributes, locking, objects seeing edits, readers racing a writer, and
h5dump (plus h5rs check in ci-test.sh) on every edited file. The
read-vs-h5py suite also runs on a file opened 'r+'.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-27 06:57:36 -05:00
co-authored by Claude Opus 5.5
parent 910d81904c
commit 92fb0830e0
15 changed files with 1978 additions and 102 deletions
Download Patch File Download Diff File Expand all files Collapse all files
crates/clawhdf5-py/src/attrs.rs
+152 -55
View File
@@ -1,23 +1,62 @@
//! PyAttrs — dict-like access to HDF5 attributes.
use std::sync::{Arc, Mutex};
use std::sync::{Arc, Mutex, PoisonError};
use clawhdf5_format::attribute::AttributeMessage;
use pyo3::exceptions::{PyKeyError, PyTypeError, PyValueError};
use pyo3::exceptions::{PyKeyError, PyNotImplementedError, PyTypeError, PyValueError};
use pyo3::prelude::*;
use pyo3::types::{PyList, PyTuple};
use crate::convert::{Converter, Elements, resolve_vl};
use crate::handle::Handle;
use crate::{OwnedAttrValue, PyEmpty, attr_value_to_py, node, py_to_attr_value};
use crate::{OwnedAttrValue, PyEmpty, attr_value_to_py, edit, node, py_to_attr_value};
/// The attributes of an object in a file opened for reading (or editing).
struct ReadAttrs {
handle: Arc<Handle>,
addr: u64,
path: String,
/// Sorted by name, with the file generation they were read at: an edit
/// (`attrs[name] = value`, here or through another handle on the same
/// object) makes them re-read.
cache: Mutex<(u64, Arc<Vec<AttributeMessage>>)>,
}
impl ReadAttrs {
fn current(&self, py: Python<'_>) -> PyResult<Arc<Vec<AttributeMessage>>> {
let generation = self.handle.generation();
{
let cached = self.cache.lock().unwrap_or_else(PoisonError::into_inner);
if cached.0 == generation {
return Ok(Arc::clone(&cached.1));
}
}
let (addr, path) = (self.addr, &self.path);
let attrs = Arc::new(self.handle.with(py, |f| node::attributes(f, addr, path))?);
*self.cache.lock().unwrap_or_else(PoisonError::into_inner) =
(generation, Arc::clone(&attrs));
Ok(attrs)
}
fn check_writable(&self) -> PyResult<()> {
if self.handle.is_writable() {
return Ok(());
}
Err(PyErr::new::<pyo3::exceptions::PyIOError, _>(
"cannot set attributes on a read-only file (open it with mode 'r+')",
))
}
fn set(&self, py: Python<'_>, name: &str, value: clawhdf5_rs::AttrValue) -> PyResult<()> {
self.check_writable()?;
let path = node::name(&self.path);
self.handle.edit(py, |ed| ed.set_attr(&path, name, &value))
}
}
/// Backing storage for attributes.
enum AttrsInner {
/// Attributes of an object in a file opened for reading, sorted by name.
Read {
handle: Arc<Handle>,
attrs: Vec<AttributeMessage>,
},
Read(ReadAttrs),
/// Writable attribute list shared with a parent (PyFile or PyGroup).
Write(Arc<Mutex<Vec<(String, OwnedAttrValue)>>>),
}
@@ -27,8 +66,11 @@ enum AttrsInner {
/// In read mode, values are what h5py returns: numpy scalars for scalar
/// attributes, numpy arrays otherwise, `str` for variable-length strings,
/// `numpy.bytes_` for fixed-length ones, and `Empty` for a null dataspace.
/// In write mode, attributes set here are accumulated and written when
/// the parent file is closed.
/// In a file opened with `'r+'`, `attrs[name] = value` adds or replaces an
/// attribute in the file at once (as h5py stores it, except that `str`
/// values become fixed-length UTF-8 strings). In write mode (`'w'`),
/// attributes set here are accumulated and written when the parent file is
/// closed.
#[pyclass(name = "Attrs")]
pub struct PyAttrs {
inner: AttrsInner,
@@ -43,9 +85,15 @@ impl PyAttrs {
addr: u64,
path: &str,
) -> PyResult<Self> {
let attrs = handle.with(py, |f| node::attributes(f, addr, path))?;
let generation = handle.generation();
let attrs = Arc::new(handle.with(py, |f| node::attributes(f, addr, path))?);
Ok(Self {
inner: AttrsInner::Read { handle, attrs },
inner: AttrsInner::Read(ReadAttrs {
handle,
addr,
path: path.to_string(),
cache: Mutex::new((generation, attrs)),
}),
})
}
@@ -55,14 +103,48 @@ impl PyAttrs {
inner: AttrsInner::Write(store),
}
}
fn set_value(
&self,
py: Python<'_>,
key: &str,
value: &Bound<'_, PyAny>,
dtype: Option<&Bound<'_, PyAny>>,
shape: Option<&Bound<'_, PyAny>>,
) -> PyResult<()> {
match &self.inner {
AttrsInner::Read(r) => {
r.check_writable()?;
let value = edit::attr_value(py, value, dtype, shape)?;
r.set(py, key, value)
}
AttrsInner::Write(store) => {
if dtype.is_some() || shape.is_some() {
return Err(PyNotImplementedError::new_err(
"attrs.create with a dtype or shape is only supported in a file opened \
with 'r+'",
));
}
let owned = py_to_attr_value(value)?;
let mut guard = store.lock().unwrap();
// Replace existing key if present.
if let Some(entry) = guard.iter_mut().find(|(k, _)| k == key) {
entry.1 = owned;
} else {
guard.push((key.to_string(), owned));
}
Ok(())
}
}
}
}
#[pymethods]
impl PyAttrs {
fn __getitem__(&self, py: Python<'_>, key: &str) -> PyResult<Py<PyAny>> {
match &self.inner {
AttrsInner::Read { handle, attrs } => match attrs.iter().find(|a| a.name == key) {
Some(attr) => Ok(attr_to_py(py, handle, attr)?.unbind()),
AttrsInner::Read(r) => match r.current(py)?.iter().find(|a| a.name == key) {
Some(attr) => Ok(attr_to_py(py, &r.handle, attr)?.unbind()),
None => Err(PyKeyError::new_err(format!(
"Can't open attribute (can't locate attribute: '{key}')"
))),
@@ -80,36 +162,63 @@ impl PyAttrs {
}
}
fn __setitem__(&self, key: &str, value: &Bound<'_, PyAny>) -> PyResult<()> {
/// `attrs[name] = value`. In a file opened with `'r+'` this writes the
/// attribute (numeric, bool, complex, bytes and str data, any shape)
/// into the file before returning; see the class docs.
fn __setitem__(&self, py: Python<'_>, key: &str, value: &Bound<'_, PyAny>) -> PyResult<()> {
self.set_value(py, key, value, None, None)
}
/// Deleting attributes is not supported in a file (the in-place editor
/// cannot remove them); in write mode it removes a pending attribute.
fn __delitem__(&self, key: &str) -> PyResult<()> {
match &self.inner {
AttrsInner::Read { .. } => Err(PyErr::new::<pyo3::exceptions::PyIOError, _>(
"cannot set attributes on a read-only file",
)),
AttrsInner::Read(_) => Err(PyNotImplementedError::new_err(format!(
"cannot delete attribute '{key}': deleting attributes is not supported by \
clawhdf5's in-place editor"
))),
AttrsInner::Write(store) => {
let owned = py_to_attr_value(value)?;
let mut guard = store.lock().unwrap();
// Replace existing key if present.
if let Some(entry) = guard.iter_mut().find(|(k, _)| k == key) {
entry.1 = owned;
} else {
guard.push((key.to_string(), owned));
let before = guard.len();
guard.retain(|(k, _)| k != key);
if guard.len() == before {
return Err(PyKeyError::new_err(key.to_string()));
}
Ok(())
}
}
}
fn __len__(&self) -> usize {
/// h5py's `attrs.create(name, data, shape=None, dtype=None)`: `data`
/// converted to `dtype` and reshaped to `shape` first.
#[pyo3(signature = (name, data, shape=None, dtype=None))]
fn create(
&self,
py: Python<'_>,
name: &str,
data: &Bound<'_, PyAny>,
shape: Option<&Bound<'_, PyAny>>,
dtype: Option<&Bound<'_, PyAny>>,
) -> PyResult<()> {
self.set_value(py, name, data, dtype, shape)
}
/// h5py's `attrs.modify(name, value)`: same as `attrs[name] = value`.
fn modify(&self, py: Python<'_>, name: &str, value: &Bound<'_, PyAny>) -> PyResult<()> {
self.set_value(py, name, value, None, None)
}
fn __len__(&self, py: Python<'_>) -> PyResult<usize> {
match &self.inner {
AttrsInner::Read { attrs, .. } => attrs.len(),
AttrsInner::Write(store) => store.lock().unwrap().len(),
AttrsInner::Read(r) => Ok(r.current(py)?.len()),
AttrsInner::Write(store) => Ok(store.lock().unwrap().len()),
}
}
fn __contains__(&self, key: &str) -> bool {
fn __contains__(&self, py: Python<'_>, key: &str) -> PyResult<bool> {
match &self.inner {
AttrsInner::Read { attrs, .. } => attrs.iter().any(|a| a.name == key),
AttrsInner::Write(store) => store.lock().unwrap().iter().any(|(k, _)| k == key),
AttrsInner::Read(r) => Ok(r.current(py)?.iter().any(|a| a.name == key)),
AttrsInner::Write(store) => Ok(store.lock().unwrap().iter().any(|(k, _)| k == key)),
}
}
@@ -119,15 +228,17 @@ impl PyAttrs {
Ok(iter)
}
fn __repr__(&self) -> String {
let n = self.__len__();
format!("<HDF5 Attrs ({n} members)>")
fn __repr__(&self, py: Python<'_>) -> String {
match self.__len__(py) {
Ok(n) => format!("<HDF5 Attrs ({n} members)>"),
Err(_) => "<HDF5 Attrs>".to_string(),
}
}
/// The value of `key`, or `default` if there is no such attribute.
#[pyo3(signature = (key, default=None))]
fn get(&self, py: Python<'_>, key: &str, default: Option<Py<PyAny>>) -> PyResult<Py<PyAny>> {
if self.__contains__(key) {
if self.__contains__(py, key)? {
self.__getitem__(py, key)
} else {
Ok(default.unwrap_or_else(|| py.None()))
@@ -137,7 +248,7 @@ impl PyAttrs {
/// Return attribute names as a list.
fn keys(&self, py: Python<'_>) -> PyResult<Py<PyAny>> {
let names: Vec<String> = match &self.inner {
AttrsInner::Read { attrs, .. } => attrs.iter().map(|a| a.name.clone()).collect(),
AttrsInner::Read(r) => r.current(py)?.iter().map(|a| a.name.clone()).collect(),
AttrsInner::Write(store) => store
.lock()
.unwrap()
@@ -152,9 +263,10 @@ impl PyAttrs {
/// Return attribute values as a list.
fn values(&self, py: Python<'_>) -> PyResult<Py<PyAny>> {
let vals: Vec<Py<PyAny>> = match &self.inner {
AttrsInner::Read { handle, attrs } => attrs
AttrsInner::Read(r) => r
.current(py)?
.iter()
.map(|a| attr_to_py(py, handle, a).map(Bound::unbind))
.map(|a| attr_to_py(py, &r.handle, a).map(Bound::unbind))
.collect::<PyResult<_>>()?,
AttrsInner::Write(store) => store
.lock()
@@ -173,9 +285,10 @@ impl PyAttrs {
/// Return attribute (key, value) pairs as a list of tuples.
fn items(&self, py: Python<'_>) -> PyResult<Py<PyAny>> {
let pairs: Vec<(String, Py<PyAny>)> = match &self.inner {
AttrsInner::Read { handle, attrs } => attrs
AttrsInner::Read(r) => r
.current(py)?
.iter()
.map(|a| Ok((a.name.clone(), attr_to_py(py, handle, a)?.unbind())))
.map(|a| Ok((a.name.clone(), attr_to_py(py, &r.handle, a)?.unbind())))
.collect::<PyResult<_>>()?,
AttrsInner::Write(store) => store
.lock()
@@ -248,19 +361,3 @@ fn prefix_err(py: Python<'_>, name: &str, e: PyErr) -> PyErr {
PyValueError::new_err(msg)
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn write_attrs_len() {
let store = Arc::new(Mutex::new(Vec::new()));
store
.lock()
.unwrap()
.push(("key".into(), OwnedAttrValue::I64(99)));
let attrs = PyAttrs::from_write(store);
assert_eq!(attrs.__len__(), 1);
}
}