clawhdf5/research/05-robustness-enhancements.md

Robustness Enhancement Recommendations

Research brief — generated 2026-08-12

---

1. Fuzzing Coverage Gaps

1.1 Current State

Two cargo-fuzz targets exist:

• fuzz_filter_pipeline — exercises the compression/decompression pipeline with arbitrary filter sequences
• fuzz_dataset_read — walks every dataset in a parsed file, exercises contiguous/chunked/compact read paths (new in unreleased work; found and fixed 3 real crash bugs)

1.2 Gaps

Write path fuzzing — the write path (FileBuilder, write_string_dataset, fractal heap construction) has no fuzz target. A malformed MemoryConfig or a corrupted in-flight write could panic or produce an invalid HDF5 file.

Recommended target:

// fuzz/fuzz_targets/fuzz_file_write.rs
#![no_main]
use libfuzzer_sys::fuzz_target;
use clawhdf5_format::{FileWriter, DatasetDescriptor};

fuzz_target!(|data: &[u8]| {
    // Interpret arbitrary bytes as a sequence of "write operations" via a
    // structured fuzzer (e.g., arbitrary::Arbitrary derive) and exercise
    // the write path into an in-memory buffer.
    let _ = exercise_write_path(data);
});

WAL replay fuzzing — the WAL has CRC32 checks and length caps (MAX_WAL_FIELD_LEN), but there is no fuzz target that feeds arbitrary byte sequences into the WAL replay path. A fuzzer here would verify that the CRC32 check correctly short-circuits before any allocation on all malformed inputs.

Knowledge graph fuzzing — the entity/relation graph accepts arbitrary strings for entity names and relation types. While these go through Rust string handling (no SQL injection possible), deeply nested graph traversal with cycles should be fuzz-tested.

Estimated effort: 1–2 days per target. Corpus from existing test fixtures.

---

2. Bounds-Check Audit Completion

2.1 Current State

The unreleased work includes a partial audit of chunked_read.rs, data_read.rs, and local_heap.rs. Three real crash bugs were fixed:

1. Integer-multiply overflow in copy_chunk_to_output's N-D assembly path
2. ndims - 1 underflow for zero-dimension chunked layouts
3. Overflow in local_heap.rs

An additional set of fixes covered:

• Paged Fixed Array: 1 << max_nelmts_bits shift overflow for u8 >= 64
• H5S selection decoder: rank capped at 32
• VDS mapping parser: no pre-allocation from untrusted nused
• Scale-offset / N-Bit: several arithmetic overflows

2.2 Remaining Work

The ROADMAP documents: "a full manual audit of every indexing site is still open."

Specific areas to audit:

• crates/clawhdf5-format/src/btree_v2.rs — B-tree v2 offset arithmetic
• crates/clawhdf5-format/src/fractal_heap.rs — heap block size calculations when building multi-direct-block heaps
• crates/clawhdf5-format/src/superblock.rs — superblock v4 (page-buffer mode) page index arithmetic
• crates/clawhdf5-format/src/extensible_array.rs — if/when extensible array support is added

Recommended approach: Use a systematic ensure_len / checked_add / checked_mul pass across all files that do offset + size arithmetic on untrusted values. The ensure_len helper already exists in the codebase — apply it everywhere it's missing.

---

3. Error Handling Improvements

3.1 Panic Sites

Rust panics on integer overflow (in debug) and silently wraps (in release without overflow-checks = true). The cargo profile should set overflow-checks = true for the format crate even in release builds, since it parses untrusted data.

Recommended addition to Cargo.toml (workspace or per-crate):

[profile.release]
overflow-checks = true  # for clawhdf5-format

Note: This may have a small performance cost (~2–5% on arithmetic-heavy code). Measure with Criterion before committing.

3.2 unwrap() / expect() in Non-Test Code

A systematic scan of non-test unwrap() calls in clawhdf5-format and clawhdf5-agent would surface latent panic sites. Recommended:

grep -rn '\.unwrap()\|\.expect(' crates/clawhdf5-format/src/ crates/clawhdf5-agent/src/ \
  | grep -v '#\[cfg(test)\]' | grep -v '// safe:'

Each hit should either be replaced with ? / explicit error handling or documented with a // SAFETY: comment explaining why the unwrap is guaranteed.

3.3 Recursive Descent Depth Guards

The CHANGELOG notes a recursion-depth guard was added for cyclic B-trees. Similar guards should exist for:

• Fractal heap traversal (if an indirect block points to itself)
• N-Bit type tree recursion (already guarded per CHANGELOG)
• Knowledge graph BFS (the bfs_neighbors function already takes a depth parameter, but the maximum depth should be explicitly capped and an error returned rather than silently truncating)

---

4. WAL Robustness

4.1 Current State

• CRC32 trailer per entry (WAL_VERSION 2)
• Length-prefix caps (MAX_WAL_FIELD_LEN = 64 MiB)
• Old-format WAL files (VERSION 1) still read and migrated on next open

4.2 Gaps

Atomic WAL rotation: If the process is killed during a WAL flush (not replay), the HDF5 file may be inconsistent with the partially-flushed WAL. The current design relies on CRC32 to detect partial entries, but the boundary between "flushed to WAL" and "committed to HDF5" is not atomic.

Recommendation: Add an explicit "commit marker" entry to the WAL (a zero-length entry with a specific magic byte sequence). The HDF5 flush marks the WAL as fully committed only after the file fsync. On replay, entries after the last commit marker are discarded.

WAL file size growth: The WAL file grows unboundedly until flush_wal() is called. A long-running agent that never flushes will accumulate a large WAL, making replay slow on restart.

Recommendation: Add an auto-flush trigger when WAL size exceeds a configurable threshold (MemoryConfig::max_wal_bytes). Default: 64 MiB.

WAL encryption: WAL entries contain plaintext memory chunks (potentially sensitive). If encryption at rest is added (see security document), the WAL should be encrypted too.

---

5. Knowledge Graph Robustness

5.1 Current State

• BFS traversal with configurable depth
• Spreading activation with configurable decay
• Fuzzy entity resolution (Levenshtein ≤ configurable distance)
• Cycle detection: the CHANGELOG mentions a "recursion-depth guard against cyclic B-trees" in the format layer, but the knowledge graph's BFS does not have an explicit cycle guard

5.2 Recommendations

Explicit cycle guard in BFS: Add a visited: HashSet<EntityId> to bfs_neighbors and spreading_activation to prevent infinite loops if a cycle exists in the graph (which is structurally possible with bidirectional relations).

Graph consistency checks on load: When loading the knowledge graph from HDF5, verify that all relation_srcs and relation_tgts reference valid entity indices. A corrupted HDF5 file could have relations pointing to nonexistent entities, causing out-of-bounds access.

Entity count cap: The knowledge graph grows unboundedly. Add a configurable max_entities and max_relations cap to prevent unbounded memory growth in long-running agents.

---

6. Multi-Modal Memory Robustness

6.1 Media Reference Storage

MediaRef stores path/URL/inline data with MIME types and FNV-1a checksums. Potential issues:

• Path traversal: If a MediaRef::Path is stored by an adversarial source and later resolved by the agent, a ../../../etc/passwd-style path could be followed. The agent should canonicalize and sandbox media paths.
• URL validation: MediaRef::Url URLs are stored as strings. An adversarial memory could store a file:// or data: URL that an agent might follow.
• Inline data size: MediaRef::Inline(Vec<u8>) has no size cap. An adversarial source could store gigabytes of inline media.

Recommendations:

1. Add MAX_INLINE_MEDIA_BYTES cap (e.g., 10 MiB).
2. Validate MediaRef::Url against an allowlist of schemes (https:// only by default).
3. Canonicalize and validate MediaRef::Path against a configurable sandbox directory.

---

7. Cross-Platform / Embedded Robustness

7.1 no_std Stability

The CHANGELOG notes that the no_std CI check was not actually running until recently (stale package names silently no-op'd the check). Now that it runs, the thumbv7em-none-eabihf build should be exercised in CI on every merge.

7.2 Endianness

HDF5 stores data in the file's native byte order (specified per-dataset). ClawHDF5 handles byte swapping for integers and floats. Verify that the following are also byte-swapped correctly:

• f16 (half-precision) values — the half crate handles this, but confirm the endianness field in the datatype message is respected
• Compound type members — each member can have a different byte order

7.3 Android JNI

The CHANGELOG documents bounds-check additions for JNI functions. Additional considerations:

• Null JNI env pointer: The JNI env pointer could theoretically be null in edge cases on older Android versions. Add a null check.
• Thread safety: JNI functions may be called from multiple Java threads. The underlying HDF5Memory uses &mut self, which is not thread-safe without external synchronization. The JNI bridge should either wrap in a Mutex or document that calls must be serialized.

---

8. Test Coverage Gaps

8.1 Integration Test Gaps

• No test exercises a full round-trip through the Python bindings with data validation
• No test exercises the Node.js bindings
• No test exercises the Android JNI bridge (these would require an Android emulator)

8.2 Property-Based Testing

The codebase uses #[cfg(test)] unit tests extensively. Adding property-based tests using proptest or quickcheck would cover:

• Round-trip invariant: write(data).then(read) == data for all valid data shapes
• Compression invariant: decompress(compress(data)) == data for all codec/data combinations
• WAL invariant: replay(wal_entries) == original_state for all valid entry sequences

Estimated effort: 1–2 weeks to add proptest to the format and agent crates with meaningful generators.

---

Priority Matrix

Item	Impact	Effort	Priority
Hybrid weight default fix	High	Trivial	P0 (also in performance doc)
overflow-checks = true in release	High	Trivial	P0
WAL auto-flush size trigger	Medium	Low	P1
Cycle guard in knowledge graph BFS	Medium	Low	P1
WAL write fuzzing target	High	Low	P1
unwrap() audit	Medium	Medium	P2
Persistent BM25 index	Medium	Medium	P2
Media reference sandboxing	Medium	Medium	P2
proptest round-trip invariants	High	Medium	P2
WAL atomic rotation / commit marker	High	High	P3
WAL encryption	High	High	P3 (blocked on encryption feature)
Graph consistency check on load	Medium	Low	P3