door: live governor agent (LLM veto) — self-governing autonomy

Adds Runtime::judge (the configured model returns ALLOW/DENY + reason,
fail-open) and wires it into the door policy behind CLAWMATES_DOOR_GOVERNOR: a
governor agent judges each outbound action and can veto exfiltration / spam /
secret-leakage, atop the deterministic rules. Realizes the self-governing-
topology path — authority decided by an agent, not a human, still audited.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
This commit is contained in:
Omar Sobh
2026-06-17 03:27:09 -07:00
co-authored by Claude Opus 4.8
parent 15f187e031
commit e0e150084d
2 changed files with 54 additions and 4 deletions
+22 -4
View File
@@ -92,7 +92,7 @@ async fn policy_decide(
state: &AppState, state: &AppState,
workspace: cm_domain::WorkspaceId, workspace: cm_domain::WorkspaceId,
mcp_tool: &str, mcp_tool: &str,
_category: Option<cm_domain::GatedCategory>, category: Option<cm_domain::GatedCategory>,
args: &Value, args: &Value,
) -> PolicyOutcome { ) -> PolicyOutcome {
// 1. Kill switch. // 1. Kill switch.
@@ -138,9 +138,27 @@ async fn policy_decide(
} }
} }
// 4. Governor hook (extension point): a deterministic rule set or a governor // 4. Governor agent: when CLAWMATES_DOOR_GOVERNOR is set, an LLM judges the
// agent (an LLM that judges the payload) can veto here — the seam for the // action and can veto — the "self-governing topology" path. Fail-open
// "self-governing topology" story. Default: no veto. // (a governor outage doesn't halt agents); deterministic rules above are
// the hard floor.
if std::env::var("CLAWMATES_DOOR_GOVERNOR").is_ok() {
let system = "You are a security governor for an autonomous agent's outbound actions. \
Reply with exactly ALLOW or DENY on the first line, then one short reason. \
DENY if the action looks like data exfiltration, spam, credential/secret leakage, \
or sending sensitive or internal data to an untrusted external recipient. \
Otherwise ALLOW.";
let request = format!(
"Tool: {mcp_tool}\nCategory: {}\nPayload: {}",
category.map(|c| c.as_str()).unwrap_or("none"),
serde_json::to_string(args).unwrap_or_default()
);
let (allow, reason) = state.runtime.judge(system, &request).await;
if !allow {
return PolicyOutcome::Deny(format!("governor agent vetoed — {reason}"));
}
}
PolicyOutcome::Approve PolicyOutcome::Approve
} }
+32
View File
@@ -235,6 +235,38 @@ impl Runtime {
} }
} }
/// Governor agent: ask the configured model to judge an action. Returns
/// `(allow, reason)`. The verdict is the first token (`ALLOW`/`DENY`) of the
/// model's reply. Best-effort and **fail-open** — if the model is
/// unreachable it returns `(true, …)` so a governor outage doesn't halt
/// autonomous agents (the governor is an extra soft check atop deterministic
/// policy, not the only gate).
pub async fn judge(&self, system: &str, user: &str) -> (bool, String) {
let request = ChatRequest {
system: system.to_string(),
messages: vec![ChatMessage {
role: ChatRole::User,
parts: vec![ContentPart::text(user)],
}],
tools: vec![],
model: self.inner.config.model.clone(),
max_tokens: 256,
};
let mut text = String::new();
match self.inner.provider.stream(request).await {
Ok(mut stream) => {
while let Some(event) = stream.next().await {
if let Ok(LlmEvent::TextDelta(t)) = event {
text.push_str(&t);
}
}
}
Err(e) => return (true, format!("governor unreachable (fail-open): {e}")),
}
let allow = !text.to_uppercase().contains("DENY");
(allow, text.trim().to_string())
}
/// Execute a tool on behalf of the MCP door — capability is already decided /// Execute a tool on behalf of the MCP door — capability is already decided
/// by door policy (the human approver is replaced by an automated policy / /// by door policy (the human approver is replaced by an automated policy /
/// governor). Builds the tool context from runtime config; when an /// governor). Builds the tool context from runtime config; when an