clawhdf5/research/06-security-hardening.md

Security Audit & Hardening Recommendations

Research brief — generated 2026-08-12

---

1. Threat Model

ClawHDF5 operates in two distinct threat environments:

Environment A — Untrusted HDF5 files: A user opens an HDF5 file from an untrusted source (downloaded file, network stream, user upload). The format parser must not crash, OOM, or execute arbitrary code.

Environment B — Agent memory under adversarial input: An AI agent writes memories sourced from external tool output, web content, or multi-agent messages. An adversary may attempt to poison the memory store by injecting crafted content.

Out of scope (by design): Network security (ClawHDF5 is a file-based library with no built-in networking). Authentication and access control at the OS level.

---

2. Current Security Posture

2.1 What's Already Done (Strong)

Control	Implementation	Coverage
Decompression output bound	MAX_DECOMPRESS_SIZE in filters.rs	Deflate, LZ4, Zstd, Pcodec
Allocation guards before alloc	Length-prefix caps before Vec::with_capacity calls	WAL (MAX_WAL_FIELD_LEN = 64 MiB), VDS mapping parser, H5S decoder
Arithmetic overflow guards	ensure_len helper; checked_add / checked_mul in critical paths	chunked_read.rs, btree_v1.rs, local_heap.rs, scale-offset, N-Bit
Recursion depth guard	Depth counter on cyclic B-tree traversal; N-Bit type tree cap	btree_v1.rs, filters.rs
WAL entry integrity	CRC32 trailer per entry (WAL_VERSION 2) — bit-flip stops replay cleanly	clawhdf5-agent::wal
Content hashing	FNV-1a for memory chunks (anomaly detection); SHA-256 for provenance attributes	provenance.rs
Injection pattern detection	15 patterns in anomaly.rs	Prompt injection, role impersonation, etc.
Write rate limiting	anomaly.rs rate limiter	Flood attacks on memory store
Source isolation	Per-MemorySource sub-stores	User vs System vs Tool source separation
Android JNI safety	Bounds-check on embedding_len; null pointer rejection	clawhdf5-android JNI functions
PyO3 safety	pyo3/numpy 0.29 (clears two RUSTSEC advisories)	Python bindings
Fuzz coverage	fuzz_filter_pipeline, fuzz_dataset_read	Filter pipeline; dataset read paths

2.2 Documented Limitations

The CHANGELOG explicitly documents:

"The integrity hashes in clawhdf5-agent::provenance (FNV-1a) and clawhdf5-format::provenance (SHA-256) are unkeyed and detect only accidental corruption, not tampering — doc-only change, no behavior change."

This is an important honesty note: the current provenance system is not a tamper-detection mechanism.

---

3. Security Gaps & Recommendations

3.1 Missing: Encryption at Rest (HIGH PRIORITY)

Gap: There is no encryption for the HDF5 file or WAL. A .brain file or agent_memory.h5 containing personal data, credentials mentioned in conversation, or proprietary knowledge is stored in plaintext.

Attack scenario: An attacker with filesystem access to the .h5 file (e.g., via a directory traversal vulnerability in an app using ClawHDF5, or physical access to a laptop) can read all agent memories.

Recommendation:

Implement an encryption feature using aes-gcm (from the aes-gcm crate — pure Rust, audited):

// Proposed API addition to MemoryConfig:
pub struct MemoryConfig {
    // ... existing fields ...
    pub encryption_key: Option<[u8; 32]>,  // AES-256-GCM key
}

Implementation approach:

1. Store a random 96-bit nonce per HDF5 chunk alongside the chunk data.
2. Encrypt each chunk's decompressed data with AES-256-GCM before writing; decrypt on read.
3. Encrypt WAL entries with the same key.
4. Store a key-derivation salt in the file header; derive the working key from a user passphrase via Argon2id.
5. The HDF5 file is still structurally valid (h5py can open it and see dataset shapes) but all data values are ciphertext — this is a deliberate tradeoff (vs encrypting the entire file as a blob).

Alternative: Encrypt the entire .h5 file as a blob using AES-256-CTR with a random IV stored in a plaintext header. Simpler but loses partial-decryption ability.

Effort estimate: 2–3 weeks. The aes-gcm and argon2 crates are well-audited and integrate cleanly into Rust.

---

3.2 Missing: Tamper Detection / Signing (HIGH PRIORITY for .brain files)

Gap: The SHA-256 provenance attributes detect accidental corruption but not intentional tampering. An adversary who can write to the .h5 file can update both the data and the SHA-256 hash.

Attack scenario: A compromised .brain file is distributed from ClawBrainHub. A user downloads it, trusting the provenance hashes, but the hashes have been re-computed over poisoned data.

Recommendation:

1. Ed25519 signatures: Add an [package] signing_key field to MemoryConfig. When signing is enabled, compute an Ed25519 signature over the dataset contents + SHA-256 provenance hash and store it as an HDF5 attribute. Verify on open.
2. ClawBrainHub trust chain: The registry should sign .brain files with a registry key. ClawHDF5 should ship a clawhdf5-cli verify command that checks the registry signature.

Crates: ed25519-dalek (pure Rust, widely audited).

Effort estimate: 1–2 weeks for basic file signing. ClawBrainHub registry integration is a separate effort.

---

3.3 Incomplete: Embedding-Space Poisoning Detection (MEDIUM PRIORITY)

Gap: The 15 injection patterns in anomaly.rs detect text-level injection attempts (e.g., "Ignore previous instructions"). They do not detect embedding-space poisoning — adversarially crafted embeddings that are semantically close to arbitrary queries in vector space but contain malicious text.

Attack scenario (from MemoryGraft paper): A tool output contains text that, when embedded, produces a vector close to "user preferences" in the embedding space. Future queries for "user preferences" retrieve the poisoned memory instead of genuine ones.

Recommendation:

1. Embedding anomaly detection: Compute the distribution of embeddings in the store (mean + covariance). Flag new embeddings whose Mahalanobis distance from the distribution centroid exceeds a threshold. This is a statistical outlier detector.
2. Cluster consistency check: After every write batch, verify that the new embedding does not shift the cluster assignment of nearby memories by more than a configurable fraction.
3. Source-aware embedding validation: Embeddings from untrusted sources (e.g., MemorySource::Tool) should be quarantined and require explicit promotion to the main store.

Effort estimate: 1–2 weeks for Mahalanobis detection; 2–3 weeks for cluster consistency.

---

3.4 Incomplete: Timestamp Integrity (MEDIUM PRIORITY)

Gap: Memory timestamps are stored in the HDF5 file as plain f64 values. The WAL CRC32 detects accidental bit-flips but not intentional timestamp manipulation by an adversary who writes to the HDF5 file.

Attack scenario: An adversary modifies timestamps in the HDF5 file to make recent poisoned memories appear old (and thus trusted by the temporal re-ranking component) or to make old poisoned memories appear recent.

Recommendation:

1. Signed timestamps: When file signing is enabled (see 3.2), include timestamps in the signed data.
2. Monotonic timestamp enforcement: In the write path, reject any attempt to write a timestamp older than the last written timestamp in the same source channel. The WAL's append-only nature already provides this for WAL entries; extend it to the HDF5 dataset.

---

3.5 JNI Thread Safety (MEDIUM PRIORITY)

Gap: The Android JNI functions operate on a raw *mut HDF5Memory handle with no synchronization. The handle is cast from a jlong and used as &mut HDF5Memory.

Attack scenario: Two Java threads call JNI functions on the same handle simultaneously → data race → undefined behavior in unsafe Rust.

Recommendation:

Wrap the HDF5Memory handle in a Mutex<HDF5Memory> and store the Mutex in a Box (as is standard for JNI handle storage):

// Current:
let memory = unsafe { &mut *(handle as *mut HDF5Memory) };

// Recommended:
let locked = unsafe { &*(handle as *const Mutex<HDF5Memory>) };
let mut memory = locked.lock().unwrap();

Effort estimate: 1–2 days. Low risk, high impact for multi-threaded Android use.

---

3.6 Media Reference Sandboxing (MEDIUM PRIORITY)

Gap: MediaRef::Path stores filesystem paths from arbitrary sources (including adversarial memory content). If the agent resolves these paths, a crafted ../../../etc/passwd path could expose sensitive files.

Recommendation:

1. Allowlist-based path validation: The agent should only resolve MediaRef::Path entries that are within a configured media_sandbox_dir.
2. Canonicalization before resolution: Always call std::fs::canonicalize before resolving a path, then check it is within the sandbox.
3. URL scheme allowlist: MediaRef::Url should only allow https:// by default. Reject file://, data:, javascript:, etc.

---

3.7 SZIP FFI Safety (LOW PRIORITY)

Gap: The szip feature introduces libaec C FFI. Incorrect FFI arguments (wrong chunk_size, mismatched bits_per_sample) could cause the C library to write past the allocated output buffer.

Recommendation:

1. The current implementation validates cd.len() >= 5 and checks bits_per_sample > 0 && <= 32. Add a check that chunk_size is non-zero and does not exceed a maximum (e.g., 512 MiB).
2. Consider wrapping the aec_buffer_decode call in std::panic::catch_unwind (if the C library signals errors via signals, not return codes — verify with libaec docs).
3. Add a fuzz target (fuzz_szip_decompress) when the szip feature is enabled.

---

3.8 Denial of Service: Adversarial HDF5 Files (LOW PRIORITY — partially mitigated)

Current mitigations: MAX_DECOMPRESS_SIZE, allocation guards, recursion depth caps, H5S_MAX_RANK cap. These collectively address the most dangerous DoS vectors.

Remaining gaps:

1. Large group with many dense links: A group with millions of links in the v2 B-tree will take O(N) memory to iterate. Add a cap (MAX_LINKS_PER_GROUP) that returns an error rather than allocating unboundedly.
2. Very long string attributes: The local_heap.rs fixes guard overflow arithmetic but there is no explicit cap on total string heap size. Add MAX_STRING_HEAP_BYTES.
3. Deeply nested compound types: The N-Bit type tree recursion is now capped (CHANGELOG), but compound types can also be nested arbitrarily. Verify compound type recursion depth is capped.

---

4. Dependency Security

4.1 RUSTSEC Advisories

The pyo3/numpy bump (0.28 → 0.29) cleared two RUSTSEC advisories. Recommended:

• Add cargo-audit to CI: cargo audit --deny warnings after every dependency update.
• Pin a cargo-audit version in CI to prevent false positives from advisory DB updates.

4.2 Supply Chain

Dependency	Risk Level	Notes
libaec-sys / libaec (SZIP)	Medium	C FFI; optional. Pin to a specific libaec version in the sys crate.
system-zlib / zlib-ng	Medium	C FFI; optional. Default path uses zlib-ng. Consider migrating to zlib-rs.
wgpu (GPU)	Low	Pure Rust + GPU driver ABI. Well-maintained.
pyo3 0.29	Low	Recently updated; audit at each bump.
tokio (async feature)	Low	Well-audited, widely used.

4.3 cargo-deny Configuration

Add deny.toml at workspace root to enforce:

• No duplicate dependencies at different semver versions
• No unmaintained crates in the dependency tree
• No licenses incompatible with MIT

---

5. Security Roadmap (Prioritized)

Item	Priority	Effort	Impact
AES-256-GCM encryption at rest	HIGH	2–3 weeks	Confidentiality for .brain / sensitive memories
Ed25519 file signing	HIGH	1–2 weeks	Tamper detection for distributed .brain files
JNI Mutex wrapping	MEDIUM	1–2 days	UB prevention on multi-threaded Android
cargo-audit in CI	MEDIUM	1 day	Continuous dependency advisory monitoring
cargo-deny configuration	LOW	1 day	Dependency hygiene
Media reference sandboxing	MEDIUM	1 week	Path traversal prevention
Embedding-space anomaly detection	MEDIUM	2–3 weeks	Poisoning resistance beyond text patterns
Monotonic timestamp enforcement	MEDIUM	3–5 days	Temporal poisoning resistance
Overflow-checks = true in release	HIGH	1 hour	Defense in depth for format parsing
WAL commit marker for atomic rotation	MEDIUM	1 week	Consistency guarantee on crash during flush
SZIP fuzz target	LOW	1 day	C FFI boundary hardening