chore: Tier 2 quick wins — version skew, docs, cleanup, overflow-safe bounds
CI / test (push) Failing after 14s

- Fix version skew: clawhdf5-py (pyproject.toml 1.93.0 -> 2.1.0) and
  packages/clawhdf5-node (package.json 2.0.0 -> 2.1.0) were both behind
  the actual crate version.
- Correct stale ROADMAP.md claims: the TypeScript bridge already has a
  complete napi-rs package (not "no package.json"); CI/CD is now wired
  up via .gitea/workflows/ci.yml.
- Fix CLAUDE.md: clawhdf5-gpu uses wgpu with hand-written WGSL compute
  shaders, not CubeCL.
- chunked_read.rs: drop 12 unnecessary chunk_dimensions[..rank].to_vec()
  allocations — all three callees already accept &[u32].
- btree_v1.rs: add an overflow-safe ensure_len(data, offset, needed)
  helper (checked_add) and use it at the two plain-arithmetic bounds
  guards, closing a usize-overflow edge case reachable from a crafted
  near-usize::MAX B-tree offset. Add a regression test.
- Clarify that the integrity hashes in clawhdf5-agent/provenance.rs
  (FNV-1a) and clawhdf5-format/provenance.rs (SHA-256) are unkeyed and
  only detect accidental corruption, not tampering — doc-only change.
- README.md: document that the mpi-io feature's read/write paths are
  root-read+broadcast / gather-to-rank-0, not true collective I/O.
This commit is contained in:
Omar Sobh
2026-08-05 12:02:23 -07:00
parent 55959b4920
commit 62595d5ac0
9 changed files with 96 additions and 45 deletions
+1 -1
View File
@@ -17,7 +17,7 @@ Cargo workspace with 16 crates under `crates/` (plus `libaec-sys`, an internal F
| `clawhdf5-netcdf4` | NetCDF-4 compatibility layer | | `clawhdf5-netcdf4` | NetCDF-4 compatibility layer |
| `clawhdf5-ann` | HNSW approximate nearest-neighbor vector index | | `clawhdf5-ann` | HNSW approximate nearest-neighbor vector index |
| `clawhdf5-agent` | Agent memory, session history, knowledge graph storage | | `clawhdf5-agent` | Agent memory, session history, knowledge graph storage |
| `clawhdf5-gpu` | GPU-accelerated I/O via CubeCL | | `clawhdf5-gpu` | GPU-accelerated I/O via wgpu (hand-written WGSL compute shaders) |
| `clawhdf5-accel` | CPU SIMD acceleration path | | `clawhdf5-accel` | CPU SIMD acceleration path |
| `clawhdf5-migrate` | Schema migration engine | | `clawhdf5-migrate` | Schema migration engine |
| `clawhdf5-android` | Android JNI bindings | | `clawhdf5-android` | Android JNI bindings |
+12
View File
@@ -420,6 +420,18 @@ ClawhDF5's agent memory design draws from 15+ recent papers:
| `system-zlib` / `zlib-rs` | no | Alternative zlib backends for deflate | | `system-zlib` / `zlib-rs` | no | Alternative zlib backends for deflate |
| `blake3_hash` | no | BLAKE3 content hashing for provenance | | `blake3_hash` | no | BLAKE3 content hashing for provenance |
### `clawhdf5-io`
| Flag | Default | Description |
|------|---------|-------------|
| `mpi-io` | no | MPI-backed I/O via the `mpi` crate |
> **Parallel I/O (MPI) limitation:** `mpi-io`'s read path is a root-rank read
> followed by a broadcast, and its write path gathers all ranks' shards to
> rank 0 before writing — not true collective I/O
> (`MPI_File_read_at_all`/`write_at_all`). It does not provide I/O bandwidth
> that scales with rank count; true collective I/O is tracked as future work.
--- ---
## Building ## Building
+8 -3
View File
@@ -151,14 +151,19 @@ All 8 tracks delivered. 1,546 tests passing, zero clippy warnings.
## What's Next ## What's Next
Verified against current repo state on 2026-08-03 (see also `docs/superpowers/plans/` for the filter-codec/format-write/MPI-IO work, now shipped): Verified against current repo state on 2026-08-05 (see also `docs/superpowers/plans/` for the filter-codec/format-write/MPI-IO work, now shipped):
- [ ] CI/CD pipeline — still no GitHub/Gitea Actions workflow in the repo; automated testing is manual only
- [ ] Academic benchmark cross-validation — reproduce MemX/LongMemEval under identical conditions - [ ] Academic benchmark cross-validation — reproduce MemX/LongMemEval under identical conditions
- [ ] TypeScript bridge — `clawhdf5-napi` has no `package.json`; it's still Rust-only scaffolding, not a publishable npm package - [ ] TypeScript bridge not wired into CI — `packages/clawhdf5-node/` already has a complete, working napi-rs package (package.json, tsconfig, hand-written TS wrapper matching all 21 `#[napi]` items, Jest test suite, README); it isn't published to npm and has no committed lockfile
- [ ] Publish crates to crates.io — no `publish` config anywhere in the workspace yet - [ ] Publish crates to crates.io — no `publish` config anywhere in the workspace yet
- [ ] Python wheel distribution via maturin — `crates/clawhdf5-py/pyproject.toml` exists (maturin-buildable locally) but wheels aren't published anywhere - [ ] Python wheel distribution via maturin — `crates/clawhdf5-py/pyproject.toml` exists (maturin-buildable locally) but wheels aren't published anywhere
### Recently closed out (2026-08-05 hardening pass)
- [x] CI/CD pipeline — `.gitea/workflows/ci.yml` now runs `scripts/ci-test.sh` (fmt, clippy, tests, no_std check) on push/PR to `main`
- [x] Fixed no_std build breakage in `clawhdf5-format` (missing alloc imports, `AtomicU64` unsupported on thumbv7em, `f64::powi` requiring std/libm)
- [x] Fixed version skew: `clawhdf5-py` (pyproject.toml) and `packages/clawhdf5-node` (package.json) were both behind the actual crate version
### Recently closed out (2026-08-03 cleanup pass) ### Recently closed out (2026-08-03 cleanup pass)
- [x] Removed `clawhdf5-types` — it was an empty 1-line stub crate; shared type definitions already live in `clawhdf5-format`, so CLAUDE.md and the workspace manifest were corrected instead of filling it in - [x] Removed `clawhdf5-types` — it was an empty 1-line stub crate; shared type definitions already live in `clawhdf5-format`, so CLAUDE.md and the workspace manifest were corrected instead of filling it in
+13 -2
View File
@@ -1,7 +1,9 @@
//! Memory provenance tracking and integrity verification. //! Memory provenance tracking and integrity verification.
//! //!
//! Records the origin, authorship, and integrity of every memory chunk //! Records the origin, authorship, and a content hash of every memory chunk
//! so the system can detect tampering and trace data lineage. //! so the system can detect *accidental* corruption and trace data lineage.
//! The hash is unkeyed (see [`fnv1a_64`]) — this is not a tamper-evidence or
//! authenticity guarantee.
use std::collections::HashMap; use std::collections::HashMap;
@@ -11,6 +13,10 @@ pub use crate::consolidation::MemorySource;
// Hash helper (std-only FNV-1a 64-bit) // Hash helper (std-only FNV-1a 64-bit)
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
/// Unkeyed, non-cryptographic FNV-1a hash for detecting accidental content
/// corruption. It is trivially forgeable by anyone able to modify the stored
/// data, since they can recompute and overwrite the stored hash alongside
/// it — do not rely on this as a tamper-evidence or authenticity control.
fn fnv1a_64(text: &str) -> u64 { fn fnv1a_64(text: &str) -> u64 {
const OFFSET: u64 = 14_695_981_039_346_656_037; const OFFSET: u64 = 14_695_981_039_346_656_037;
const PRIME: u64 = 1_099_511_628_211; const PRIME: u64 = 1_099_511_628_211;
@@ -114,6 +120,11 @@ impl ProvenanceStore {
/// Re-hash `current_chunk` and compare against the stored hash. /// Re-hash `current_chunk` and compare against the stored hash.
/// Returns `true` if the content matches (integrity intact). /// Returns `true` if the content matches (integrity intact).
///
/// This only detects accidental corruption: the hash is unkeyed, so an
/// actor able to modify the stored chunk can also recompute and
/// overwrite the stored hash. Do not treat a `true` result as proof the
/// data hasn't been tampered with.
pub fn verify_integrity(&self, record_id: u64, current_chunk: &str) -> bool { pub fn verify_integrity(&self, record_id: u64, current_chunk: &str) -> bool {
match self.records.get(&record_id) { match self.records.get(&record_id) {
Some(p) => p.content_hash == fnv1a_64(current_chunk), Some(p) => p.content_hash == fnv1a_64(current_chunk),
+28 -13
View File
@@ -24,6 +24,21 @@ pub struct BTreeV1Node {
pub children: Vec<u64>, pub children: Vec<u64>,
} }
/// Checks that `[offset, offset + needed)` fits within `data`, guarding the
/// addition against `usize` overflow from a crafted near-`usize::MAX` offset.
fn ensure_len(data: &[u8], offset: usize, needed: usize) -> Result<(), FormatError> {
if offset
.checked_add(needed)
.is_none_or(|end| end > data.len())
{
return Err(FormatError::UnexpectedEof {
expected: offset.saturating_add(needed),
available: data.len(),
});
}
Ok(())
}
fn read_offset(data: &[u8], pos: usize, size: u8) -> Result<u64, FormatError> { fn read_offset(data: &[u8], pos: usize, size: u8) -> Result<u64, FormatError> {
let s = size as usize; let s = size as usize;
if pos.checked_add(s).is_none_or(|end| end > data.len()) { if pos.checked_add(s).is_none_or(|end| end > data.len()) {
@@ -45,7 +60,7 @@ fn read_offset(data: &[u8], pos: usize, size: u8) -> Result<u64, FormatError> {
fn is_undefined(data: &[u8], pos: usize, size: u8) -> bool { fn is_undefined(data: &[u8], pos: usize, size: u8) -> bool {
let s = size as usize; let s = size as usize;
if pos + s > data.len() { if ensure_len(data, pos, s).is_err() {
return false; return false;
} }
data[pos..pos + s].iter().all(|&b| b == 0xFF) data[pos..pos + s].iter().all(|&b| b == 0xFF)
@@ -65,12 +80,7 @@ impl BTreeV1Node {
// + left_sibling(offset_size) + right_sibling(offset_size) // + left_sibling(offset_size) + right_sibling(offset_size)
let os = offset_size as usize; let os = offset_size as usize;
let header_size = 8 + os * 2; let header_size = 8 + os * 2;
if offset + header_size > file_data.len() { ensure_len(file_data, offset, header_size)?;
return Err(FormatError::UnexpectedEof {
expected: offset + header_size,
available: file_data.len(),
});
}
if &file_data[offset..offset + 4] != b"TREE" { if &file_data[offset..offset + 4] != b"TREE" {
return Err(FormatError::InvalidBTreeSignature); return Err(FormatError::InvalidBTreeSignature);
@@ -99,12 +109,7 @@ impl BTreeV1Node {
let eu = entries_used as usize; let eu = entries_used as usize;
let key_size = os; // For type 0, key = offset_size let key_size = os; // For type 0, key = offset_size
let needed = eu * (key_size + os) + key_size; // eu children + (eu+1) keys let needed = eu * (key_size + os) + key_size; // eu children + (eu+1) keys
if pos + needed > file_data.len() { ensure_len(file_data, pos, needed)?;
return Err(FormatError::UnexpectedEof {
expected: pos + needed,
available: file_data.len(),
});
}
let mut keys = Vec::with_capacity(eu + 1); let mut keys = Vec::with_capacity(eu + 1);
let mut children = Vec::with_capacity(eu); let mut children = Vec::with_capacity(eu);
@@ -241,6 +246,16 @@ mod tests {
assert_eq!(node.right_sibling, None); assert_eq!(node.right_sibling, None);
} }
#[test]
fn parse_near_usize_max_offset_rejected_without_overflow() {
let data = build_btree_node(0, 0, &[0, 5, 10], &[0x100, 0x200], None, None, 8);
let result = BTreeV1Node::parse(&data, usize::MAX - 4, 8, 8);
assert!(
matches!(result, Err(FormatError::UnexpectedEof { .. })),
"expected a clean UnexpectedEof, got {result:?}"
);
}
#[test] #[test]
fn parse_with_siblings_none() { fn parse_with_siblings_none() {
let data = build_btree_node(0, 0, &[0, 8], &[0x300], None, None, 8); let data = build_btree_node(0, 0, &[0, 8], &[0x300], None, None, 8);
+24 -24
View File
@@ -386,24 +386,24 @@ pub fn read_chunked_data(
} }
(4, Some(2)) => { (4, Some(2)) => {
// Implicit index — use spatial chunk dims only // Implicit index — use spatial chunk dims only
let spatial_chunk_dims: Vec<u32> = chunk_dimensions[..rank].to_vec(); let spatial_chunk_dims: &[u32] = &chunk_dimensions[..rank];
generate_implicit_chunks( generate_implicit_chunks(
addr, addr,
&dataspace.dimensions, &dataspace.dimensions,
&spatial_chunk_dims, spatial_chunk_dims,
elem_size as u32, elem_size as u32,
) )
} }
(4, Some(3)) => { (4, Some(3)) => {
// Fixed Array — use spatial chunk dims only // Fixed Array — use spatial chunk dims only
let spatial_chunk_dims: Vec<u32> = chunk_dimensions[..rank].to_vec(); let spatial_chunk_dims: &[u32] = &chunk_dimensions[..rank];
let header = let header =
FixedArrayHeader::parse(file_data, addr as usize, offset_size, length_size)?; FixedArrayHeader::parse(file_data, addr as usize, offset_size, length_size)?;
read_fixed_array_chunks( read_fixed_array_chunks(
file_data, file_data,
&header, &header,
&dataspace.dimensions, &dataspace.dimensions,
&spatial_chunk_dims, spatial_chunk_dims,
elem_size as u32, elem_size as u32,
offset_size, offset_size,
length_size, length_size,
@@ -411,14 +411,14 @@ pub fn read_chunked_data(
} }
(4, Some(4)) => { (4, Some(4)) => {
// Extensible Array — use spatial chunk dims only // Extensible Array — use spatial chunk dims only
let spatial_chunk_dims: Vec<u32> = chunk_dimensions[..rank].to_vec(); let spatial_chunk_dims: &[u32] = &chunk_dimensions[..rank];
let header = let header =
ExtensibleArrayHeader::parse(file_data, addr as usize, offset_size, length_size)?; ExtensibleArrayHeader::parse(file_data, addr as usize, offset_size, length_size)?;
read_extensible_array_chunks( read_extensible_array_chunks(
file_data, file_data,
&header, &header,
&dataspace.dimensions, &dataspace.dimensions,
&spatial_chunk_dims, spatial_chunk_dims,
elem_size as u32, elem_size as u32,
offset_size, offset_size,
length_size, length_size,
@@ -618,30 +618,30 @@ pub fn read_chunked_data_cached(
}] }]
} }
(4, Some(2)) => { (4, Some(2)) => {
let spatial_chunk_dims: Vec<u32> = chunk_dimensions[..rank].to_vec(); let spatial_chunk_dims: &[u32] = &chunk_dimensions[..rank];
generate_implicit_chunks( generate_implicit_chunks(
addr, addr,
&dataspace.dimensions, &dataspace.dimensions,
&spatial_chunk_dims, spatial_chunk_dims,
elem_size as u32, elem_size as u32,
) )
} }
(4, Some(3)) => { (4, Some(3)) => {
let spatial_chunk_dims: Vec<u32> = chunk_dimensions[..rank].to_vec(); let spatial_chunk_dims: &[u32] = &chunk_dimensions[..rank];
let header = let header =
FixedArrayHeader::parse(file_data, addr as usize, offset_size, length_size)?; FixedArrayHeader::parse(file_data, addr as usize, offset_size, length_size)?;
read_fixed_array_chunks( read_fixed_array_chunks(
file_data, file_data,
&header, &header,
&dataspace.dimensions, &dataspace.dimensions,
&spatial_chunk_dims, spatial_chunk_dims,
elem_size as u32, elem_size as u32,
offset_size, offset_size,
length_size, length_size,
)? )?
} }
(4, Some(4)) => { (4, Some(4)) => {
let spatial_chunk_dims: Vec<u32> = chunk_dimensions[..rank].to_vec(); let spatial_chunk_dims: &[u32] = &chunk_dimensions[..rank];
let header = ExtensibleArrayHeader::parse( let header = ExtensibleArrayHeader::parse(
file_data, file_data,
addr as usize, addr as usize,
@@ -652,7 +652,7 @@ pub fn read_chunked_data_cached(
file_data, file_data,
&header, &header,
&dataspace.dimensions, &dataspace.dimensions,
&spatial_chunk_dims, spatial_chunk_dims,
elem_size as u32, elem_size as u32,
offset_size, offset_size,
length_size, length_size,
@@ -974,30 +974,30 @@ pub fn read_chunked_data_sweep(
}] }]
} }
(4, Some(2)) => { (4, Some(2)) => {
let spatial_chunk_dims: Vec<u32> = chunk_dimensions[..rank].to_vec(); let spatial_chunk_dims: &[u32] = &chunk_dimensions[..rank];
generate_implicit_chunks( generate_implicit_chunks(
addr, addr,
&dataspace.dimensions, &dataspace.dimensions,
&spatial_chunk_dims, spatial_chunk_dims,
elem_size as u32, elem_size as u32,
) )
} }
(4, Some(3)) => { (4, Some(3)) => {
let spatial_chunk_dims: Vec<u32> = chunk_dimensions[..rank].to_vec(); let spatial_chunk_dims: &[u32] = &chunk_dimensions[..rank];
let header = let header =
FixedArrayHeader::parse(file_data, addr as usize, offset_size, length_size)?; FixedArrayHeader::parse(file_data, addr as usize, offset_size, length_size)?;
read_fixed_array_chunks( read_fixed_array_chunks(
file_data, file_data,
&header, &header,
&dataspace.dimensions, &dataspace.dimensions,
&spatial_chunk_dims, spatial_chunk_dims,
elem_size as u32, elem_size as u32,
offset_size, offset_size,
length_size, length_size,
)? )?
} }
(4, Some(4)) => { (4, Some(4)) => {
let spatial_chunk_dims: Vec<u32> = chunk_dimensions[..rank].to_vec(); let spatial_chunk_dims: &[u32] = &chunk_dimensions[..rank];
let header = ExtensibleArrayHeader::parse( let header = ExtensibleArrayHeader::parse(
file_data, file_data,
addr as usize, addr as usize,
@@ -1008,7 +1008,7 @@ pub fn read_chunked_data_sweep(
file_data, file_data,
&header, &header,
&dataspace.dimensions, &dataspace.dimensions,
&spatial_chunk_dims, spatial_chunk_dims,
elem_size as u32, elem_size as u32,
offset_size, offset_size,
length_size, length_size,
@@ -1200,30 +1200,30 @@ pub fn read_chunked_data_indexed(
}] }]
} }
(4, Some(2)) => { (4, Some(2)) => {
let spatial_chunk_dims: Vec<u32> = chunk_dimensions[..rank].to_vec(); let spatial_chunk_dims: &[u32] = &chunk_dimensions[..rank];
generate_implicit_chunks( generate_implicit_chunks(
addr, addr,
&dataspace.dimensions, &dataspace.dimensions,
&spatial_chunk_dims, spatial_chunk_dims,
elem_size as u32, elem_size as u32,
) )
} }
(4, Some(3)) => { (4, Some(3)) => {
let spatial_chunk_dims: Vec<u32> = chunk_dimensions[..rank].to_vec(); let spatial_chunk_dims: &[u32] = &chunk_dimensions[..rank];
let header = let header =
FixedArrayHeader::parse(file_data, addr as usize, offset_size, length_size)?; FixedArrayHeader::parse(file_data, addr as usize, offset_size, length_size)?;
read_fixed_array_chunks( read_fixed_array_chunks(
file_data, file_data,
&header, &header,
&dataspace.dimensions, &dataspace.dimensions,
&spatial_chunk_dims, spatial_chunk_dims,
elem_size as u32, elem_size as u32,
offset_size, offset_size,
length_size, length_size,
)? )?
} }
(4, Some(4)) => { (4, Some(4)) => {
let spatial_chunk_dims: Vec<u32> = chunk_dimensions[..rank].to_vec(); let spatial_chunk_dims: &[u32] = &chunk_dimensions[..rank];
let header = ExtensibleArrayHeader::parse( let header = ExtensibleArrayHeader::parse(
file_data, file_data,
addr as usize, addr as usize,
@@ -1234,7 +1234,7 @@ pub fn read_chunked_data_indexed(
file_data, file_data,
&header, &header,
&dataspace.dimensions, &dataspace.dimensions,
&spatial_chunk_dims, spatial_chunk_dims,
elem_size as u32, elem_size as u32,
offset_size, offset_size,
length_size, length_size,
+8
View File
@@ -2,6 +2,9 @@
//! data-integrity verification. //! data-integrity verification.
//! //!
//! Enable with the `provenance` Cargo feature (on by default). //! Enable with the `provenance` Cargo feature (on by default).
//!
//! The hash is unkeyed, so this detects accidental corruption only — it is
//! not a tamper-evidence or authenticity guarantee. See [`verify_dataset`].
#[cfg(not(feature = "std"))] #[cfg(not(feature = "std"))]
use alloc::{format, string::String, vec::Vec}; use alloc::{format, string::String, vec::Vec};
@@ -115,6 +118,11 @@ pub enum VerifyResult {
/// ///
/// `file_data` is the entire HDF5 file bytes; `header` is the parsed object /// `file_data` is the entire HDF5 file bytes; `header` is the parsed object
/// header for the dataset of interest. /// header for the dataset of interest.
///
/// This only detects *accidental* corruption. The hash is unkeyed and stored
/// alongside the data it protects, so anyone able to modify the dataset can
/// also recompute and overwrite `_provenance_sha256` — a `VerifyResult::Ok`
/// is not a tamper-evidence or authenticity guarantee.
pub fn verify_dataset( pub fn verify_dataset(
file_data: &[u8], file_data: &[u8],
header: &ObjectHeader, header: &ObjectHeader,
+1 -1
View File
@@ -4,7 +4,7 @@ build-backend = "maturin"
[project] [project]
name = "rustyhdf5" name = "rustyhdf5"
version = "1.93.0" version = "2.1.0"
description = "Python bindings for rustyhdf5 — a pure-Rust HDF5 library" description = "Python bindings for rustyhdf5 — a pure-Rust HDF5 library"
requires-python = ">=3.8" requires-python = ">=3.8"
license = { text = "MIT" } license = { text = "MIT" }
+1 -1
View File
@@ -1,6 +1,6 @@
{ {
"name": "@redclaw/clawhdf5", "name": "@redclaw/clawhdf5",
"version": "2.0.0", "version": "2.1.0",
"description": "Node.js bindings for clawhdf5 — HDF5-backed agent memory with hippocampal consolidation", "description": "Node.js bindings for clawhdf5 — HDF5-backed agent memory with hippocampal consolidation",
"main": "index.js", "main": "index.js",
"types": "index.d.ts", "types": "index.d.ts",