door: live governor agent (LLM veto) — self-governing autonomy
Adds Runtime::judge (the configured model returns ALLOW/DENY + reason, fail-open) and wires it into the door policy behind CLAWMATES_DOOR_GOVERNOR: a governor agent judges each outbound action and can veto exfiltration / spam / secret-leakage, atop the deterministic rules. Realizes the self-governing- topology path — authority decided by an agent, not a human, still audited. Co-Authored-By: Claude Opus 4.8 <[email protected]>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
15f187e031
commit
e0e150084d
@@ -92,7 +92,7 @@ async fn policy_decide(
|
|||||||
state: &AppState,
|
state: &AppState,
|
||||||
workspace: cm_domain::WorkspaceId,
|
workspace: cm_domain::WorkspaceId,
|
||||||
mcp_tool: &str,
|
mcp_tool: &str,
|
||||||
_category: Option<cm_domain::GatedCategory>,
|
category: Option<cm_domain::GatedCategory>,
|
||||||
args: &Value,
|
args: &Value,
|
||||||
) -> PolicyOutcome {
|
) -> PolicyOutcome {
|
||||||
// 1. Kill switch.
|
// 1. Kill switch.
|
||||||
@@ -138,9 +138,27 @@ async fn policy_decide(
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// 4. Governor hook (extension point): a deterministic rule set or a governor
|
// 4. Governor agent: when CLAWMATES_DOOR_GOVERNOR is set, an LLM judges the
|
||||||
// agent (an LLM that judges the payload) can veto here — the seam for the
|
// action and can veto — the "self-governing topology" path. Fail-open
|
||||||
// "self-governing topology" story. Default: no veto.
|
// (a governor outage doesn't halt agents); deterministic rules above are
|
||||||
|
// the hard floor.
|
||||||
|
if std::env::var("CLAWMATES_DOOR_GOVERNOR").is_ok() {
|
||||||
|
let system = "You are a security governor for an autonomous agent's outbound actions. \
|
||||||
|
Reply with exactly ALLOW or DENY on the first line, then one short reason. \
|
||||||
|
DENY if the action looks like data exfiltration, spam, credential/secret leakage, \
|
||||||
|
or sending sensitive or internal data to an untrusted external recipient. \
|
||||||
|
Otherwise ALLOW.";
|
||||||
|
let request = format!(
|
||||||
|
"Tool: {mcp_tool}\nCategory: {}\nPayload: {}",
|
||||||
|
category.map(|c| c.as_str()).unwrap_or("none"),
|
||||||
|
serde_json::to_string(args).unwrap_or_default()
|
||||||
|
);
|
||||||
|
let (allow, reason) = state.runtime.judge(system, &request).await;
|
||||||
|
if !allow {
|
||||||
|
return PolicyOutcome::Deny(format!("governor agent vetoed — {reason}"));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
PolicyOutcome::Approve
|
PolicyOutcome::Approve
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -235,6 +235,38 @@ impl Runtime {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Governor agent: ask the configured model to judge an action. Returns
|
||||||
|
/// `(allow, reason)`. The verdict is the first token (`ALLOW`/`DENY`) of the
|
||||||
|
/// model's reply. Best-effort and **fail-open** — if the model is
|
||||||
|
/// unreachable it returns `(true, …)` so a governor outage doesn't halt
|
||||||
|
/// autonomous agents (the governor is an extra soft check atop deterministic
|
||||||
|
/// policy, not the only gate).
|
||||||
|
pub async fn judge(&self, system: &str, user: &str) -> (bool, String) {
|
||||||
|
let request = ChatRequest {
|
||||||
|
system: system.to_string(),
|
||||||
|
messages: vec![ChatMessage {
|
||||||
|
role: ChatRole::User,
|
||||||
|
parts: vec![ContentPart::text(user)],
|
||||||
|
}],
|
||||||
|
tools: vec![],
|
||||||
|
model: self.inner.config.model.clone(),
|
||||||
|
max_tokens: 256,
|
||||||
|
};
|
||||||
|
let mut text = String::new();
|
||||||
|
match self.inner.provider.stream(request).await {
|
||||||
|
Ok(mut stream) => {
|
||||||
|
while let Some(event) = stream.next().await {
|
||||||
|
if let Ok(LlmEvent::TextDelta(t)) = event {
|
||||||
|
text.push_str(&t);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
Err(e) => return (true, format!("governor unreachable (fail-open): {e}")),
|
||||||
|
}
|
||||||
|
let allow = !text.to_uppercase().contains("DENY");
|
||||||
|
(allow, text.trim().to_string())
|
||||||
|
}
|
||||||
|
|
||||||
/// Execute a tool on behalf of the MCP door — capability is already decided
|
/// Execute a tool on behalf of the MCP door — capability is already decided
|
||||||
/// by door policy (the human approver is replaced by an automated policy /
|
/// by door policy (the human approver is replaced by an automated policy /
|
||||||
/// governor). Builds the tool context from runtime config; when an
|
/// governor). Builds the tool context from runtime config; when an
|
||||||
|
|||||||
Reference in New Issue
Block a user