chore: Tier 2 quick wins — version skew, docs, cleanup, overflow-safe bounds
CI / test (push) Failing after 14s
CI / test (push) Failing after 14s
- Fix version skew: clawhdf5-py (pyproject.toml 1.93.0 -> 2.1.0) and packages/clawhdf5-node (package.json 2.0.0 -> 2.1.0) were both behind the actual crate version. - Correct stale ROADMAP.md claims: the TypeScript bridge already has a complete napi-rs package (not "no package.json"); CI/CD is now wired up via .gitea/workflows/ci.yml. - Fix CLAUDE.md: clawhdf5-gpu uses wgpu with hand-written WGSL compute shaders, not CubeCL. - chunked_read.rs: drop 12 unnecessary chunk_dimensions[..rank].to_vec() allocations — all three callees already accept &[u32]. - btree_v1.rs: add an overflow-safe ensure_len(data, offset, needed) helper (checked_add) and use it at the two plain-arithmetic bounds guards, closing a usize-overflow edge case reachable from a crafted near-usize::MAX B-tree offset. Add a regression test. - Clarify that the integrity hashes in clawhdf5-agent/provenance.rs (FNV-1a) and clawhdf5-format/provenance.rs (SHA-256) are unkeyed and only detect accidental corruption, not tampering — doc-only change. - README.md: document that the mpi-io feature's read/write paths are root-read+broadcast / gather-to-rank-0, not true collective I/O.
This commit is contained in:
@@ -17,7 +17,7 @@ Cargo workspace with 16 crates under `crates/` (plus `libaec-sys`, an internal F
|
|||||||
| `clawhdf5-netcdf4` | NetCDF-4 compatibility layer |
|
| `clawhdf5-netcdf4` | NetCDF-4 compatibility layer |
|
||||||
| `clawhdf5-ann` | HNSW approximate nearest-neighbor vector index |
|
| `clawhdf5-ann` | HNSW approximate nearest-neighbor vector index |
|
||||||
| `clawhdf5-agent` | Agent memory, session history, knowledge graph storage |
|
| `clawhdf5-agent` | Agent memory, session history, knowledge graph storage |
|
||||||
| `clawhdf5-gpu` | GPU-accelerated I/O via CubeCL |
|
| `clawhdf5-gpu` | GPU-accelerated I/O via wgpu (hand-written WGSL compute shaders) |
|
||||||
| `clawhdf5-accel` | CPU SIMD acceleration path |
|
| `clawhdf5-accel` | CPU SIMD acceleration path |
|
||||||
| `clawhdf5-migrate` | Schema migration engine |
|
| `clawhdf5-migrate` | Schema migration engine |
|
||||||
| `clawhdf5-android` | Android JNI bindings |
|
| `clawhdf5-android` | Android JNI bindings |
|
||||||
|
|||||||
@@ -420,6 +420,18 @@ ClawhDF5's agent memory design draws from 15+ recent papers:
|
|||||||
| `system-zlib` / `zlib-rs` | no | Alternative zlib backends for deflate |
|
| `system-zlib` / `zlib-rs` | no | Alternative zlib backends for deflate |
|
||||||
| `blake3_hash` | no | BLAKE3 content hashing for provenance |
|
| `blake3_hash` | no | BLAKE3 content hashing for provenance |
|
||||||
|
|
||||||
|
### `clawhdf5-io`
|
||||||
|
|
||||||
|
| Flag | Default | Description |
|
||||||
|
|------|---------|-------------|
|
||||||
|
| `mpi-io` | no | MPI-backed I/O via the `mpi` crate |
|
||||||
|
|
||||||
|
> **Parallel I/O (MPI) limitation:** `mpi-io`'s read path is a root-rank read
|
||||||
|
> followed by a broadcast, and its write path gathers all ranks' shards to
|
||||||
|
> rank 0 before writing — not true collective I/O
|
||||||
|
> (`MPI_File_read_at_all`/`write_at_all`). It does not provide I/O bandwidth
|
||||||
|
> that scales with rank count; true collective I/O is tracked as future work.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Building
|
## Building
|
||||||
|
|||||||
+8
-3
@@ -151,14 +151,19 @@ All 8 tracks delivered. 1,546 tests passing, zero clippy warnings.
|
|||||||
|
|
||||||
## What's Next
|
## What's Next
|
||||||
|
|
||||||
Verified against current repo state on 2026-08-03 (see also `docs/superpowers/plans/` for the filter-codec/format-write/MPI-IO work, now shipped):
|
Verified against current repo state on 2026-08-05 (see also `docs/superpowers/plans/` for the filter-codec/format-write/MPI-IO work, now shipped):
|
||||||
|
|
||||||
- [ ] CI/CD pipeline — still no GitHub/Gitea Actions workflow in the repo; automated testing is manual only
|
|
||||||
- [ ] Academic benchmark cross-validation — reproduce MemX/LongMemEval under identical conditions
|
- [ ] Academic benchmark cross-validation — reproduce MemX/LongMemEval under identical conditions
|
||||||
- [ ] TypeScript bridge — `clawhdf5-napi` has no `package.json`; it's still Rust-only scaffolding, not a publishable npm package
|
- [ ] TypeScript bridge not wired into CI — `packages/clawhdf5-node/` already has a complete, working napi-rs package (package.json, tsconfig, hand-written TS wrapper matching all 21 `#[napi]` items, Jest test suite, README); it isn't published to npm and has no committed lockfile
|
||||||
- [ ] Publish crates to crates.io — no `publish` config anywhere in the workspace yet
|
- [ ] Publish crates to crates.io — no `publish` config anywhere in the workspace yet
|
||||||
- [ ] Python wheel distribution via maturin — `crates/clawhdf5-py/pyproject.toml` exists (maturin-buildable locally) but wheels aren't published anywhere
|
- [ ] Python wheel distribution via maturin — `crates/clawhdf5-py/pyproject.toml` exists (maturin-buildable locally) but wheels aren't published anywhere
|
||||||
|
|
||||||
|
### Recently closed out (2026-08-05 hardening pass)
|
||||||
|
|
||||||
|
- [x] CI/CD pipeline — `.gitea/workflows/ci.yml` now runs `scripts/ci-test.sh` (fmt, clippy, tests, no_std check) on push/PR to `main`
|
||||||
|
- [x] Fixed no_std build breakage in `clawhdf5-format` (missing alloc imports, `AtomicU64` unsupported on thumbv7em, `f64::powi` requiring std/libm)
|
||||||
|
- [x] Fixed version skew: `clawhdf5-py` (pyproject.toml) and `packages/clawhdf5-node` (package.json) were both behind the actual crate version
|
||||||
|
|
||||||
### Recently closed out (2026-08-03 cleanup pass)
|
### Recently closed out (2026-08-03 cleanup pass)
|
||||||
|
|
||||||
- [x] Removed `clawhdf5-types` — it was an empty 1-line stub crate; shared type definitions already live in `clawhdf5-format`, so CLAUDE.md and the workspace manifest were corrected instead of filling it in
|
- [x] Removed `clawhdf5-types` — it was an empty 1-line stub crate; shared type definitions already live in `clawhdf5-format`, so CLAUDE.md and the workspace manifest were corrected instead of filling it in
|
||||||
|
|||||||
@@ -1,7 +1,9 @@
|
|||||||
//! Memory provenance tracking and integrity verification.
|
//! Memory provenance tracking and integrity verification.
|
||||||
//!
|
//!
|
||||||
//! Records the origin, authorship, and integrity of every memory chunk
|
//! Records the origin, authorship, and a content hash of every memory chunk
|
||||||
//! so the system can detect tampering and trace data lineage.
|
//! so the system can detect *accidental* corruption and trace data lineage.
|
||||||
|
//! The hash is unkeyed (see [`fnv1a_64`]) — this is not a tamper-evidence or
|
||||||
|
//! authenticity guarantee.
|
||||||
|
|
||||||
use std::collections::HashMap;
|
use std::collections::HashMap;
|
||||||
|
|
||||||
@@ -11,6 +13,10 @@ pub use crate::consolidation::MemorySource;
|
|||||||
// Hash helper (std-only FNV-1a 64-bit)
|
// Hash helper (std-only FNV-1a 64-bit)
|
||||||
// ---------------------------------------------------------------------------
|
// ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
/// Unkeyed, non-cryptographic FNV-1a hash for detecting accidental content
|
||||||
|
/// corruption. It is trivially forgeable by anyone able to modify the stored
|
||||||
|
/// data, since they can recompute and overwrite the stored hash alongside
|
||||||
|
/// it — do not rely on this as a tamper-evidence or authenticity control.
|
||||||
fn fnv1a_64(text: &str) -> u64 {
|
fn fnv1a_64(text: &str) -> u64 {
|
||||||
const OFFSET: u64 = 14_695_981_039_346_656_037;
|
const OFFSET: u64 = 14_695_981_039_346_656_037;
|
||||||
const PRIME: u64 = 1_099_511_628_211;
|
const PRIME: u64 = 1_099_511_628_211;
|
||||||
@@ -114,6 +120,11 @@ impl ProvenanceStore {
|
|||||||
|
|
||||||
/// Re-hash `current_chunk` and compare against the stored hash.
|
/// Re-hash `current_chunk` and compare against the stored hash.
|
||||||
/// Returns `true` if the content matches (integrity intact).
|
/// Returns `true` if the content matches (integrity intact).
|
||||||
|
///
|
||||||
|
/// This only detects accidental corruption: the hash is unkeyed, so an
|
||||||
|
/// actor able to modify the stored chunk can also recompute and
|
||||||
|
/// overwrite the stored hash. Do not treat a `true` result as proof the
|
||||||
|
/// data hasn't been tampered with.
|
||||||
pub fn verify_integrity(&self, record_id: u64, current_chunk: &str) -> bool {
|
pub fn verify_integrity(&self, record_id: u64, current_chunk: &str) -> bool {
|
||||||
match self.records.get(&record_id) {
|
match self.records.get(&record_id) {
|
||||||
Some(p) => p.content_hash == fnv1a_64(current_chunk),
|
Some(p) => p.content_hash == fnv1a_64(current_chunk),
|
||||||
|
|||||||
@@ -24,6 +24,21 @@ pub struct BTreeV1Node {
|
|||||||
pub children: Vec<u64>,
|
pub children: Vec<u64>,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Checks that `[offset, offset + needed)` fits within `data`, guarding the
|
||||||
|
/// addition against `usize` overflow from a crafted near-`usize::MAX` offset.
|
||||||
|
fn ensure_len(data: &[u8], offset: usize, needed: usize) -> Result<(), FormatError> {
|
||||||
|
if offset
|
||||||
|
.checked_add(needed)
|
||||||
|
.is_none_or(|end| end > data.len())
|
||||||
|
{
|
||||||
|
return Err(FormatError::UnexpectedEof {
|
||||||
|
expected: offset.saturating_add(needed),
|
||||||
|
available: data.len(),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
fn read_offset(data: &[u8], pos: usize, size: u8) -> Result<u64, FormatError> {
|
fn read_offset(data: &[u8], pos: usize, size: u8) -> Result<u64, FormatError> {
|
||||||
let s = size as usize;
|
let s = size as usize;
|
||||||
if pos.checked_add(s).is_none_or(|end| end > data.len()) {
|
if pos.checked_add(s).is_none_or(|end| end > data.len()) {
|
||||||
@@ -45,7 +60,7 @@ fn read_offset(data: &[u8], pos: usize, size: u8) -> Result<u64, FormatError> {
|
|||||||
|
|
||||||
fn is_undefined(data: &[u8], pos: usize, size: u8) -> bool {
|
fn is_undefined(data: &[u8], pos: usize, size: u8) -> bool {
|
||||||
let s = size as usize;
|
let s = size as usize;
|
||||||
if pos + s > data.len() {
|
if ensure_len(data, pos, s).is_err() {
|
||||||
return false;
|
return false;
|
||||||
}
|
}
|
||||||
data[pos..pos + s].iter().all(|&b| b == 0xFF)
|
data[pos..pos + s].iter().all(|&b| b == 0xFF)
|
||||||
@@ -65,12 +80,7 @@ impl BTreeV1Node {
|
|||||||
// + left_sibling(offset_size) + right_sibling(offset_size)
|
// + left_sibling(offset_size) + right_sibling(offset_size)
|
||||||
let os = offset_size as usize;
|
let os = offset_size as usize;
|
||||||
let header_size = 8 + os * 2;
|
let header_size = 8 + os * 2;
|
||||||
if offset + header_size > file_data.len() {
|
ensure_len(file_data, offset, header_size)?;
|
||||||
return Err(FormatError::UnexpectedEof {
|
|
||||||
expected: offset + header_size,
|
|
||||||
available: file_data.len(),
|
|
||||||
});
|
|
||||||
}
|
|
||||||
|
|
||||||
if &file_data[offset..offset + 4] != b"TREE" {
|
if &file_data[offset..offset + 4] != b"TREE" {
|
||||||
return Err(FormatError::InvalidBTreeSignature);
|
return Err(FormatError::InvalidBTreeSignature);
|
||||||
@@ -99,12 +109,7 @@ impl BTreeV1Node {
|
|||||||
let eu = entries_used as usize;
|
let eu = entries_used as usize;
|
||||||
let key_size = os; // For type 0, key = offset_size
|
let key_size = os; // For type 0, key = offset_size
|
||||||
let needed = eu * (key_size + os) + key_size; // eu children + (eu+1) keys
|
let needed = eu * (key_size + os) + key_size; // eu children + (eu+1) keys
|
||||||
if pos + needed > file_data.len() {
|
ensure_len(file_data, pos, needed)?;
|
||||||
return Err(FormatError::UnexpectedEof {
|
|
||||||
expected: pos + needed,
|
|
||||||
available: file_data.len(),
|
|
||||||
});
|
|
||||||
}
|
|
||||||
|
|
||||||
let mut keys = Vec::with_capacity(eu + 1);
|
let mut keys = Vec::with_capacity(eu + 1);
|
||||||
let mut children = Vec::with_capacity(eu);
|
let mut children = Vec::with_capacity(eu);
|
||||||
@@ -241,6 +246,16 @@ mod tests {
|
|||||||
assert_eq!(node.right_sibling, None);
|
assert_eq!(node.right_sibling, None);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn parse_near_usize_max_offset_rejected_without_overflow() {
|
||||||
|
let data = build_btree_node(0, 0, &[0, 5, 10], &[0x100, 0x200], None, None, 8);
|
||||||
|
let result = BTreeV1Node::parse(&data, usize::MAX - 4, 8, 8);
|
||||||
|
assert!(
|
||||||
|
matches!(result, Err(FormatError::UnexpectedEof { .. })),
|
||||||
|
"expected a clean UnexpectedEof, got {result:?}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn parse_with_siblings_none() {
|
fn parse_with_siblings_none() {
|
||||||
let data = build_btree_node(0, 0, &[0, 8], &[0x300], None, None, 8);
|
let data = build_btree_node(0, 0, &[0, 8], &[0x300], None, None, 8);
|
||||||
|
|||||||
@@ -386,24 +386,24 @@ pub fn read_chunked_data(
|
|||||||
}
|
}
|
||||||
(4, Some(2)) => {
|
(4, Some(2)) => {
|
||||||
// Implicit index — use spatial chunk dims only
|
// Implicit index — use spatial chunk dims only
|
||||||
let spatial_chunk_dims: Vec<u32> = chunk_dimensions[..rank].to_vec();
|
let spatial_chunk_dims: &[u32] = &chunk_dimensions[..rank];
|
||||||
generate_implicit_chunks(
|
generate_implicit_chunks(
|
||||||
addr,
|
addr,
|
||||||
&dataspace.dimensions,
|
&dataspace.dimensions,
|
||||||
&spatial_chunk_dims,
|
spatial_chunk_dims,
|
||||||
elem_size as u32,
|
elem_size as u32,
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
(4, Some(3)) => {
|
(4, Some(3)) => {
|
||||||
// Fixed Array — use spatial chunk dims only
|
// Fixed Array — use spatial chunk dims only
|
||||||
let spatial_chunk_dims: Vec<u32> = chunk_dimensions[..rank].to_vec();
|
let spatial_chunk_dims: &[u32] = &chunk_dimensions[..rank];
|
||||||
let header =
|
let header =
|
||||||
FixedArrayHeader::parse(file_data, addr as usize, offset_size, length_size)?;
|
FixedArrayHeader::parse(file_data, addr as usize, offset_size, length_size)?;
|
||||||
read_fixed_array_chunks(
|
read_fixed_array_chunks(
|
||||||
file_data,
|
file_data,
|
||||||
&header,
|
&header,
|
||||||
&dataspace.dimensions,
|
&dataspace.dimensions,
|
||||||
&spatial_chunk_dims,
|
spatial_chunk_dims,
|
||||||
elem_size as u32,
|
elem_size as u32,
|
||||||
offset_size,
|
offset_size,
|
||||||
length_size,
|
length_size,
|
||||||
@@ -411,14 +411,14 @@ pub fn read_chunked_data(
|
|||||||
}
|
}
|
||||||
(4, Some(4)) => {
|
(4, Some(4)) => {
|
||||||
// Extensible Array — use spatial chunk dims only
|
// Extensible Array — use spatial chunk dims only
|
||||||
let spatial_chunk_dims: Vec<u32> = chunk_dimensions[..rank].to_vec();
|
let spatial_chunk_dims: &[u32] = &chunk_dimensions[..rank];
|
||||||
let header =
|
let header =
|
||||||
ExtensibleArrayHeader::parse(file_data, addr as usize, offset_size, length_size)?;
|
ExtensibleArrayHeader::parse(file_data, addr as usize, offset_size, length_size)?;
|
||||||
read_extensible_array_chunks(
|
read_extensible_array_chunks(
|
||||||
file_data,
|
file_data,
|
||||||
&header,
|
&header,
|
||||||
&dataspace.dimensions,
|
&dataspace.dimensions,
|
||||||
&spatial_chunk_dims,
|
spatial_chunk_dims,
|
||||||
elem_size as u32,
|
elem_size as u32,
|
||||||
offset_size,
|
offset_size,
|
||||||
length_size,
|
length_size,
|
||||||
@@ -618,30 +618,30 @@ pub fn read_chunked_data_cached(
|
|||||||
}]
|
}]
|
||||||
}
|
}
|
||||||
(4, Some(2)) => {
|
(4, Some(2)) => {
|
||||||
let spatial_chunk_dims: Vec<u32> = chunk_dimensions[..rank].to_vec();
|
let spatial_chunk_dims: &[u32] = &chunk_dimensions[..rank];
|
||||||
generate_implicit_chunks(
|
generate_implicit_chunks(
|
||||||
addr,
|
addr,
|
||||||
&dataspace.dimensions,
|
&dataspace.dimensions,
|
||||||
&spatial_chunk_dims,
|
spatial_chunk_dims,
|
||||||
elem_size as u32,
|
elem_size as u32,
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
(4, Some(3)) => {
|
(4, Some(3)) => {
|
||||||
let spatial_chunk_dims: Vec<u32> = chunk_dimensions[..rank].to_vec();
|
let spatial_chunk_dims: &[u32] = &chunk_dimensions[..rank];
|
||||||
let header =
|
let header =
|
||||||
FixedArrayHeader::parse(file_data, addr as usize, offset_size, length_size)?;
|
FixedArrayHeader::parse(file_data, addr as usize, offset_size, length_size)?;
|
||||||
read_fixed_array_chunks(
|
read_fixed_array_chunks(
|
||||||
file_data,
|
file_data,
|
||||||
&header,
|
&header,
|
||||||
&dataspace.dimensions,
|
&dataspace.dimensions,
|
||||||
&spatial_chunk_dims,
|
spatial_chunk_dims,
|
||||||
elem_size as u32,
|
elem_size as u32,
|
||||||
offset_size,
|
offset_size,
|
||||||
length_size,
|
length_size,
|
||||||
)?
|
)?
|
||||||
}
|
}
|
||||||
(4, Some(4)) => {
|
(4, Some(4)) => {
|
||||||
let spatial_chunk_dims: Vec<u32> = chunk_dimensions[..rank].to_vec();
|
let spatial_chunk_dims: &[u32] = &chunk_dimensions[..rank];
|
||||||
let header = ExtensibleArrayHeader::parse(
|
let header = ExtensibleArrayHeader::parse(
|
||||||
file_data,
|
file_data,
|
||||||
addr as usize,
|
addr as usize,
|
||||||
@@ -652,7 +652,7 @@ pub fn read_chunked_data_cached(
|
|||||||
file_data,
|
file_data,
|
||||||
&header,
|
&header,
|
||||||
&dataspace.dimensions,
|
&dataspace.dimensions,
|
||||||
&spatial_chunk_dims,
|
spatial_chunk_dims,
|
||||||
elem_size as u32,
|
elem_size as u32,
|
||||||
offset_size,
|
offset_size,
|
||||||
length_size,
|
length_size,
|
||||||
@@ -974,30 +974,30 @@ pub fn read_chunked_data_sweep(
|
|||||||
}]
|
}]
|
||||||
}
|
}
|
||||||
(4, Some(2)) => {
|
(4, Some(2)) => {
|
||||||
let spatial_chunk_dims: Vec<u32> = chunk_dimensions[..rank].to_vec();
|
let spatial_chunk_dims: &[u32] = &chunk_dimensions[..rank];
|
||||||
generate_implicit_chunks(
|
generate_implicit_chunks(
|
||||||
addr,
|
addr,
|
||||||
&dataspace.dimensions,
|
&dataspace.dimensions,
|
||||||
&spatial_chunk_dims,
|
spatial_chunk_dims,
|
||||||
elem_size as u32,
|
elem_size as u32,
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
(4, Some(3)) => {
|
(4, Some(3)) => {
|
||||||
let spatial_chunk_dims: Vec<u32> = chunk_dimensions[..rank].to_vec();
|
let spatial_chunk_dims: &[u32] = &chunk_dimensions[..rank];
|
||||||
let header =
|
let header =
|
||||||
FixedArrayHeader::parse(file_data, addr as usize, offset_size, length_size)?;
|
FixedArrayHeader::parse(file_data, addr as usize, offset_size, length_size)?;
|
||||||
read_fixed_array_chunks(
|
read_fixed_array_chunks(
|
||||||
file_data,
|
file_data,
|
||||||
&header,
|
&header,
|
||||||
&dataspace.dimensions,
|
&dataspace.dimensions,
|
||||||
&spatial_chunk_dims,
|
spatial_chunk_dims,
|
||||||
elem_size as u32,
|
elem_size as u32,
|
||||||
offset_size,
|
offset_size,
|
||||||
length_size,
|
length_size,
|
||||||
)?
|
)?
|
||||||
}
|
}
|
||||||
(4, Some(4)) => {
|
(4, Some(4)) => {
|
||||||
let spatial_chunk_dims: Vec<u32> = chunk_dimensions[..rank].to_vec();
|
let spatial_chunk_dims: &[u32] = &chunk_dimensions[..rank];
|
||||||
let header = ExtensibleArrayHeader::parse(
|
let header = ExtensibleArrayHeader::parse(
|
||||||
file_data,
|
file_data,
|
||||||
addr as usize,
|
addr as usize,
|
||||||
@@ -1008,7 +1008,7 @@ pub fn read_chunked_data_sweep(
|
|||||||
file_data,
|
file_data,
|
||||||
&header,
|
&header,
|
||||||
&dataspace.dimensions,
|
&dataspace.dimensions,
|
||||||
&spatial_chunk_dims,
|
spatial_chunk_dims,
|
||||||
elem_size as u32,
|
elem_size as u32,
|
||||||
offset_size,
|
offset_size,
|
||||||
length_size,
|
length_size,
|
||||||
@@ -1200,30 +1200,30 @@ pub fn read_chunked_data_indexed(
|
|||||||
}]
|
}]
|
||||||
}
|
}
|
||||||
(4, Some(2)) => {
|
(4, Some(2)) => {
|
||||||
let spatial_chunk_dims: Vec<u32> = chunk_dimensions[..rank].to_vec();
|
let spatial_chunk_dims: &[u32] = &chunk_dimensions[..rank];
|
||||||
generate_implicit_chunks(
|
generate_implicit_chunks(
|
||||||
addr,
|
addr,
|
||||||
&dataspace.dimensions,
|
&dataspace.dimensions,
|
||||||
&spatial_chunk_dims,
|
spatial_chunk_dims,
|
||||||
elem_size as u32,
|
elem_size as u32,
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
(4, Some(3)) => {
|
(4, Some(3)) => {
|
||||||
let spatial_chunk_dims: Vec<u32> = chunk_dimensions[..rank].to_vec();
|
let spatial_chunk_dims: &[u32] = &chunk_dimensions[..rank];
|
||||||
let header =
|
let header =
|
||||||
FixedArrayHeader::parse(file_data, addr as usize, offset_size, length_size)?;
|
FixedArrayHeader::parse(file_data, addr as usize, offset_size, length_size)?;
|
||||||
read_fixed_array_chunks(
|
read_fixed_array_chunks(
|
||||||
file_data,
|
file_data,
|
||||||
&header,
|
&header,
|
||||||
&dataspace.dimensions,
|
&dataspace.dimensions,
|
||||||
&spatial_chunk_dims,
|
spatial_chunk_dims,
|
||||||
elem_size as u32,
|
elem_size as u32,
|
||||||
offset_size,
|
offset_size,
|
||||||
length_size,
|
length_size,
|
||||||
)?
|
)?
|
||||||
}
|
}
|
||||||
(4, Some(4)) => {
|
(4, Some(4)) => {
|
||||||
let spatial_chunk_dims: Vec<u32> = chunk_dimensions[..rank].to_vec();
|
let spatial_chunk_dims: &[u32] = &chunk_dimensions[..rank];
|
||||||
let header = ExtensibleArrayHeader::parse(
|
let header = ExtensibleArrayHeader::parse(
|
||||||
file_data,
|
file_data,
|
||||||
addr as usize,
|
addr as usize,
|
||||||
@@ -1234,7 +1234,7 @@ pub fn read_chunked_data_indexed(
|
|||||||
file_data,
|
file_data,
|
||||||
&header,
|
&header,
|
||||||
&dataspace.dimensions,
|
&dataspace.dimensions,
|
||||||
&spatial_chunk_dims,
|
spatial_chunk_dims,
|
||||||
elem_size as u32,
|
elem_size as u32,
|
||||||
offset_size,
|
offset_size,
|
||||||
length_size,
|
length_size,
|
||||||
|
|||||||
@@ -2,6 +2,9 @@
|
|||||||
//! data-integrity verification.
|
//! data-integrity verification.
|
||||||
//!
|
//!
|
||||||
//! Enable with the `provenance` Cargo feature (on by default).
|
//! Enable with the `provenance` Cargo feature (on by default).
|
||||||
|
//!
|
||||||
|
//! The hash is unkeyed, so this detects accidental corruption only — it is
|
||||||
|
//! not a tamper-evidence or authenticity guarantee. See [`verify_dataset`].
|
||||||
|
|
||||||
#[cfg(not(feature = "std"))]
|
#[cfg(not(feature = "std"))]
|
||||||
use alloc::{format, string::String, vec::Vec};
|
use alloc::{format, string::String, vec::Vec};
|
||||||
@@ -115,6 +118,11 @@ pub enum VerifyResult {
|
|||||||
///
|
///
|
||||||
/// `file_data` is the entire HDF5 file bytes; `header` is the parsed object
|
/// `file_data` is the entire HDF5 file bytes; `header` is the parsed object
|
||||||
/// header for the dataset of interest.
|
/// header for the dataset of interest.
|
||||||
|
///
|
||||||
|
/// This only detects *accidental* corruption. The hash is unkeyed and stored
|
||||||
|
/// alongside the data it protects, so anyone able to modify the dataset can
|
||||||
|
/// also recompute and overwrite `_provenance_sha256` — a `VerifyResult::Ok`
|
||||||
|
/// is not a tamper-evidence or authenticity guarantee.
|
||||||
pub fn verify_dataset(
|
pub fn verify_dataset(
|
||||||
file_data: &[u8],
|
file_data: &[u8],
|
||||||
header: &ObjectHeader,
|
header: &ObjectHeader,
|
||||||
|
|||||||
@@ -4,7 +4,7 @@ build-backend = "maturin"
|
|||||||
|
|
||||||
[project]
|
[project]
|
||||||
name = "rustyhdf5"
|
name = "rustyhdf5"
|
||||||
version = "1.93.0"
|
version = "2.1.0"
|
||||||
description = "Python bindings for rustyhdf5 — a pure-Rust HDF5 library"
|
description = "Python bindings for rustyhdf5 — a pure-Rust HDF5 library"
|
||||||
requires-python = ">=3.8"
|
requires-python = ">=3.8"
|
||||||
license = { text = "MIT" }
|
license = { text = "MIT" }
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
{
|
{
|
||||||
"name": "@redclaw/clawhdf5",
|
"name": "@redclaw/clawhdf5",
|
||||||
"version": "2.0.0",
|
"version": "2.1.0",
|
||||||
"description": "Node.js bindings for clawhdf5 — HDF5-backed agent memory with hippocampal consolidation",
|
"description": "Node.js bindings for clawhdf5 — HDF5-backed agent memory with hippocampal consolidation",
|
||||||
"main": "index.js",
|
"main": "index.js",
|
||||||
"types": "index.d.ts",
|
"types": "index.d.ts",
|
||||||
|
|||||||
Reference in New Issue
Block a user