ClawHDF5 PlannerandClaude Sonnet 4.6
14db35aa74
Seven research briefs covering the full mission scope: 01 — Architecture overview (crate map, format coverage, agent modules) 02 — Roadmap status and strategic gaps (distribution, MPI-IO, encryption) 03 — HDF5 ecosystem and cutting-edge developments (HDF5 2.0, Blosc2, ANN trends) 04 — Performance optimizations (10 opportunities, prioritized) 05 — Robustness enhancements (fuzzing gaps, bounds audit, WAL, KG cycle guard) 06 — Security hardening (encryption, signing, embedding poisoning, JNI safety) 07 — Synthesis and 15 actionable next steps with INT-NN task markers Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
224 lines
13 KiB
Markdown
224 lines
13 KiB
Markdown
# Security Audit & Hardening Recommendations
|
||
|
||
*Research brief — generated 2026-08-12*
|
||
|
||
---
|
||
|
||
## 1. Threat Model
|
||
|
||
ClawHDF5 operates in two distinct threat environments:
|
||
|
||
**Environment A — Untrusted HDF5 files**: A user opens an HDF5 file from an untrusted source (downloaded file, network stream, user upload). The format parser must not crash, OOM, or execute arbitrary code.
|
||
|
||
**Environment B — Agent memory under adversarial input**: An AI agent writes memories sourced from external tool output, web content, or multi-agent messages. An adversary may attempt to poison the memory store by injecting crafted content.
|
||
|
||
**Out of scope (by design):** Network security (ClawHDF5 is a file-based library with no built-in networking). Authentication and access control at the OS level.
|
||
|
||
---
|
||
|
||
## 2. Current Security Posture
|
||
|
||
### 2.1 What's Already Done (Strong)
|
||
|
||
| Control | Implementation | Coverage |
|
||
|---------|----------------|----------|
|
||
| **Decompression output bound** | `MAX_DECOMPRESS_SIZE` in `filters.rs` | Deflate, LZ4, Zstd, Pcodec |
|
||
| **Allocation guards before alloc** | Length-prefix caps before `Vec::with_capacity` calls | WAL (`MAX_WAL_FIELD_LEN` = 64 MiB), VDS mapping parser, H5S decoder |
|
||
| **Arithmetic overflow guards** | `ensure_len` helper; `checked_add` / `checked_mul` in critical paths | `chunked_read.rs`, `btree_v1.rs`, `local_heap.rs`, scale-offset, N-Bit |
|
||
| **Recursion depth guard** | Depth counter on cyclic B-tree traversal; N-Bit type tree cap | `btree_v1.rs`, `filters.rs` |
|
||
| **WAL entry integrity** | CRC32 trailer per entry (WAL_VERSION 2) — bit-flip stops replay cleanly | `clawhdf5-agent::wal` |
|
||
| **Content hashing** | FNV-1a for memory chunks (anomaly detection); SHA-256 for provenance attributes | `provenance.rs` |
|
||
| **Injection pattern detection** | 15 patterns in `anomaly.rs` | Prompt injection, role impersonation, etc. |
|
||
| **Write rate limiting** | `anomaly.rs` rate limiter | Flood attacks on memory store |
|
||
| **Source isolation** | Per-`MemorySource` sub-stores | User vs System vs Tool source separation |
|
||
| **Android JNI safety** | Bounds-check on `embedding_len`; null pointer rejection | `clawhdf5-android` JNI functions |
|
||
| **PyO3 safety** | pyo3/numpy 0.29 (clears two RUSTSEC advisories) | Python bindings |
|
||
| **Fuzz coverage** | `fuzz_filter_pipeline`, `fuzz_dataset_read` | Filter pipeline; dataset read paths |
|
||
|
||
### 2.2 Documented Limitations
|
||
|
||
The CHANGELOG explicitly documents:
|
||
> "The integrity hashes in `clawhdf5-agent::provenance` (FNV-1a) and `clawhdf5-format::provenance` (SHA-256) are unkeyed and detect only accidental corruption, not tampering — doc-only change, no behavior change."
|
||
|
||
This is an important honesty note: the current provenance system is **not** a tamper-detection mechanism.
|
||
|
||
---
|
||
|
||
## 3. Security Gaps & Recommendations
|
||
|
||
### 3.1 Missing: Encryption at Rest (HIGH PRIORITY)
|
||
|
||
**Gap:** There is no encryption for the HDF5 file or WAL. A `.brain` file or `agent_memory.h5` containing personal data, credentials mentioned in conversation, or proprietary knowledge is stored in plaintext.
|
||
|
||
**Attack scenario:** An attacker with filesystem access to the `.h5` file (e.g., via a directory traversal vulnerability in an app using ClawHDF5, or physical access to a laptop) can read all agent memories.
|
||
|
||
**Recommendation:**
|
||
|
||
Implement an `encryption` feature using `aes-gcm` (from the `aes-gcm` crate — pure Rust, audited):
|
||
|
||
```rust
|
||
// Proposed API addition to MemoryConfig:
|
||
pub struct MemoryConfig {
|
||
// ... existing fields ...
|
||
pub encryption_key: Option<[u8; 32]>, // AES-256-GCM key
|
||
}
|
||
```
|
||
|
||
Implementation approach:
|
||
1. Store a random 96-bit nonce per HDF5 chunk alongside the chunk data.
|
||
2. Encrypt each chunk's decompressed data with AES-256-GCM before writing; decrypt on read.
|
||
3. Encrypt WAL entries with the same key.
|
||
4. Store a key-derivation salt in the file header; derive the working key from a user passphrase via Argon2id.
|
||
5. The HDF5 file is still structurally valid (h5py can open it and see dataset shapes) but all data values are ciphertext — this is a deliberate tradeoff (vs encrypting the entire file as a blob).
|
||
|
||
**Alternative:** Encrypt the entire `.h5` file as a blob using AES-256-CTR with a random IV stored in a plaintext header. Simpler but loses partial-decryption ability.
|
||
|
||
**Effort estimate:** 2–3 weeks. The `aes-gcm` and `argon2` crates are well-audited and integrate cleanly into Rust.
|
||
|
||
---
|
||
|
||
### 3.2 Missing: Tamper Detection / Signing (HIGH PRIORITY for `.brain` files)
|
||
|
||
**Gap:** The SHA-256 provenance attributes detect accidental corruption but not intentional tampering. An adversary who can write to the `.h5` file can update both the data and the SHA-256 hash.
|
||
|
||
**Attack scenario:** A compromised `.brain` file is distributed from ClawBrainHub. A user downloads it, trusting the provenance hashes, but the hashes have been re-computed over poisoned data.
|
||
|
||
**Recommendation:**
|
||
|
||
1. **Ed25519 signatures**: Add an `[package] signing_key` field to `MemoryConfig`. When signing is enabled, compute an Ed25519 signature over the dataset contents + SHA-256 provenance hash and store it as an HDF5 attribute. Verify on open.
|
||
2. **ClawBrainHub trust chain**: The registry should sign `.brain` files with a registry key. ClawHDF5 should ship a `clawhdf5-cli verify` command that checks the registry signature.
|
||
|
||
**Crates:** `ed25519-dalek` (pure Rust, widely audited).
|
||
|
||
**Effort estimate:** 1–2 weeks for basic file signing. ClawBrainHub registry integration is a separate effort.
|
||
|
||
---
|
||
|
||
### 3.3 Incomplete: Embedding-Space Poisoning Detection (MEDIUM PRIORITY)
|
||
|
||
**Gap:** The 15 injection patterns in `anomaly.rs` detect text-level injection attempts (e.g., "Ignore previous instructions"). They do not detect **embedding-space poisoning** — adversarially crafted embeddings that are semantically close to arbitrary queries in vector space but contain malicious text.
|
||
|
||
**Attack scenario (from MemoryGraft paper):** A tool output contains text that, when embedded, produces a vector close to "user preferences" in the embedding space. Future queries for "user preferences" retrieve the poisoned memory instead of genuine ones.
|
||
|
||
**Recommendation:**
|
||
|
||
1. **Embedding anomaly detection**: Compute the distribution of embeddings in the store (mean + covariance). Flag new embeddings whose Mahalanobis distance from the distribution centroid exceeds a threshold. This is a statistical outlier detector.
|
||
2. **Cluster consistency check**: After every write batch, verify that the new embedding does not shift the cluster assignment of nearby memories by more than a configurable fraction.
|
||
3. **Source-aware embedding validation**: Embeddings from untrusted sources (e.g., `MemorySource::Tool`) should be quarantined and require explicit promotion to the main store.
|
||
|
||
**Effort estimate:** 1–2 weeks for Mahalanobis detection; 2–3 weeks for cluster consistency.
|
||
|
||
---
|
||
|
||
### 3.4 Incomplete: Timestamp Integrity (MEDIUM PRIORITY)
|
||
|
||
**Gap:** Memory timestamps are stored in the HDF5 file as plain `f64` values. The WAL CRC32 detects accidental bit-flips but not intentional timestamp manipulation by an adversary who writes to the HDF5 file.
|
||
|
||
**Attack scenario:** An adversary modifies timestamps in the HDF5 file to make recent poisoned memories appear old (and thus trusted by the temporal re-ranking component) or to make old poisoned memories appear recent.
|
||
|
||
**Recommendation:**
|
||
|
||
1. **Signed timestamps**: When file signing is enabled (see 3.2), include timestamps in the signed data.
|
||
2. **Monotonic timestamp enforcement**: In the write path, reject any attempt to write a timestamp older than the last written timestamp in the same source channel. The WAL's append-only nature already provides this for WAL entries; extend it to the HDF5 dataset.
|
||
|
||
---
|
||
|
||
### 3.5 JNI Thread Safety (MEDIUM PRIORITY)
|
||
|
||
**Gap:** The Android JNI functions operate on a raw `*mut HDF5Memory` handle with no synchronization. The handle is cast from a `jlong` and used as `&mut HDF5Memory`.
|
||
|
||
**Attack scenario:** Two Java threads call JNI functions on the same handle simultaneously → data race → undefined behavior in unsafe Rust.
|
||
|
||
**Recommendation:**
|
||
|
||
Wrap the `HDF5Memory` handle in a `Mutex<HDF5Memory>` and store the `Mutex` in a `Box` (as is standard for JNI handle storage):
|
||
|
||
```rust
|
||
// Current:
|
||
let memory = unsafe { &mut *(handle as *mut HDF5Memory) };
|
||
|
||
// Recommended:
|
||
let locked = unsafe { &*(handle as *const Mutex<HDF5Memory>) };
|
||
let mut memory = locked.lock().unwrap();
|
||
```
|
||
|
||
**Effort estimate:** 1–2 days. Low risk, high impact for multi-threaded Android use.
|
||
|
||
---
|
||
|
||
### 3.6 Media Reference Sandboxing (MEDIUM PRIORITY)
|
||
|
||
**Gap:** `MediaRef::Path` stores filesystem paths from arbitrary sources (including adversarial memory content). If the agent resolves these paths, a crafted `../../../etc/passwd` path could expose sensitive files.
|
||
|
||
**Recommendation:**
|
||
|
||
1. **Allowlist-based path validation**: The agent should only resolve `MediaRef::Path` entries that are within a configured `media_sandbox_dir`.
|
||
2. **Canonicalization before resolution**: Always call `std::fs::canonicalize` before resolving a path, then check it is within the sandbox.
|
||
3. **URL scheme allowlist**: `MediaRef::Url` should only allow `https://` by default. Reject `file://`, `data:`, `javascript:`, etc.
|
||
|
||
---
|
||
|
||
### 3.7 SZIP FFI Safety (LOW PRIORITY)
|
||
|
||
**Gap:** The `szip` feature introduces `libaec` C FFI. Incorrect FFI arguments (wrong `chunk_size`, mismatched `bits_per_sample`) could cause the C library to write past the allocated output buffer.
|
||
|
||
**Recommendation:**
|
||
|
||
1. The current implementation validates `cd.len() >= 5` and checks `bits_per_sample > 0 && <= 32`. Add a check that `chunk_size` is non-zero and does not exceed a maximum (e.g., 512 MiB).
|
||
2. Consider wrapping the `aec_buffer_decode` call in `std::panic::catch_unwind` (if the C library signals errors via signals, not return codes — verify with libaec docs).
|
||
3. Add a fuzz target (`fuzz_szip_decompress`) when the `szip` feature is enabled.
|
||
|
||
---
|
||
|
||
### 3.8 Denial of Service: Adversarial HDF5 Files (LOW PRIORITY — partially mitigated)
|
||
|
||
**Current mitigations:** `MAX_DECOMPRESS_SIZE`, allocation guards, recursion depth caps, `H5S_MAX_RANK` cap. These collectively address the most dangerous DoS vectors.
|
||
|
||
**Remaining gaps:**
|
||
|
||
1. **Large group with many dense links**: A group with millions of links in the v2 B-tree will take O(N) memory to iterate. Add a cap (`MAX_LINKS_PER_GROUP`) that returns an error rather than allocating unboundedly.
|
||
2. **Very long string attributes**: The `local_heap.rs` fixes guard overflow arithmetic but there is no explicit cap on total string heap size. Add `MAX_STRING_HEAP_BYTES`.
|
||
3. **Deeply nested compound types**: The N-Bit type tree recursion is now capped (CHANGELOG), but compound types can also be nested arbitrarily. Verify compound type recursion depth is capped.
|
||
|
||
---
|
||
|
||
## 4. Dependency Security
|
||
|
||
### 4.1 RUSTSEC Advisories
|
||
The pyo3/numpy bump (0.28 → 0.29) cleared two RUSTSEC advisories. Recommended:
|
||
- Add `cargo-audit` to CI: `cargo audit --deny warnings` after every dependency update.
|
||
- Pin a `cargo-audit` version in CI to prevent false positives from advisory DB updates.
|
||
|
||
### 4.2 Supply Chain
|
||
| Dependency | Risk Level | Notes |
|
||
|------------|------------|-------|
|
||
| `libaec-sys` / libaec (SZIP) | Medium | C FFI; optional. Pin to a specific libaec version in the sys crate. |
|
||
| `system-zlib` / zlib-ng | Medium | C FFI; optional. Default path uses zlib-ng. Consider migrating to `zlib-rs`. |
|
||
| `wgpu` (GPU) | Low | Pure Rust + GPU driver ABI. Well-maintained. |
|
||
| `pyo3` 0.29 | Low | Recently updated; audit at each bump. |
|
||
| `tokio` (`async` feature) | Low | Well-audited, widely used. |
|
||
|
||
### 4.3 `cargo-deny` Configuration
|
||
Add `deny.toml` at workspace root to enforce:
|
||
- No duplicate dependencies at different semver versions
|
||
- No `unmaintained` crates in the dependency tree
|
||
- No licenses incompatible with MIT
|
||
|
||
---
|
||
|
||
## 5. Security Roadmap (Prioritized)
|
||
|
||
| Item | Priority | Effort | Impact |
|
||
|------|----------|--------|--------|
|
||
| AES-256-GCM encryption at rest | HIGH | 2–3 weeks | Confidentiality for `.brain` / sensitive memories |
|
||
| Ed25519 file signing | HIGH | 1–2 weeks | Tamper detection for distributed `.brain` files |
|
||
| JNI `Mutex` wrapping | MEDIUM | 1–2 days | UB prevention on multi-threaded Android |
|
||
| `cargo-audit` in CI | MEDIUM | 1 day | Continuous dependency advisory monitoring |
|
||
| `cargo-deny` configuration | LOW | 1 day | Dependency hygiene |
|
||
| Media reference sandboxing | MEDIUM | 1 week | Path traversal prevention |
|
||
| Embedding-space anomaly detection | MEDIUM | 2–3 weeks | Poisoning resistance beyond text patterns |
|
||
| Monotonic timestamp enforcement | MEDIUM | 3–5 days | Temporal poisoning resistance |
|
||
| Overflow-checks = true in release | HIGH | 1 hour | Defense in depth for format parsing |
|
||
| WAL commit marker for atomic rotation | MEDIUM | 1 week | Consistency guarantee on crash during flush |
|
||
| SZIP fuzz target | LOW | 1 day | C FFI boundary hardening |
|