cm_decide:🚪 three Nouls per outbound action (data leaving the
organisation, a credential in the content, unsolicited/abusive), the max
is the deny probability. Measured on 24 hand-labelled door actions
(eval/door-actions.json): AUROC 1.000, [email protected] 0.96, no false denies, no
misses, 4 of 24 in the review band — three deny-labelled actions it would
not refuse alone (db dump 0.71, delegate-exfil 0.74, threat 0.77) and the
one genuinely borderline allow (repo name to a contractor 0.56). 168 ms,
~600 tokens per action, off the z.ai quota.
mcp_door: PolicyOutcome::Hold. With TYPESAFE_API_KEY set, above DENY_AT
(0.8) refused, below ALLOW_BELOW (0.2) executed, between them the action
gets a pending approval (session_key door:<id>) and the agent is told it
is queued and not to retry. The approvals route recognises a held door
action and executes it on approve — the grant decide mints, the tool
consumes — rather than resuming a chat run. The chat-model governor
stays as the fallback without a key; it has no middle band. Fail-closed
on an unreachable or malformed answer. Thresholds overridable per
deployment (CLAWMATES_DOOR_DENY_AT / _ALLOW_BELOW).
decide-eval --kind door reports the band outcome, not only a threshold.
Harness: a door scenario exercising all three bands directly against /mcp
with email_send (its effect is an outbox row), then approving the held
one and checking it executes then and not before.
Co-Authored-By: Claude Opus 5 <[email protected]>
A third kind of decision-maker between deterministic code and a full LLM
call: Choice / Score / Noul questions answered as probability
distributions with a confidence, behind one Decider trait, with the
composition patterns (confidence gating, composite scoring, rerank) as
code. Two backends: TypeSafe's Jev over HTTP, and a DeBERTa-v3 MNLI
cross-encoder run in-process with candle (feature nli; metal/cuda).
decide-eval measures a backend on labelled cases the way judge-eval
measures the judge. eval/skill-triage.json: 20 mission tasks × 53 skills,
75 positives, hand-labelled. Measured 2026-09-21:
lexical overlap AUROC 0.851 [email protected] 0.47 top-k 48/75 ECE 0.095
jev (named wording) AUROC 0.989 [email protected] 0.84 top-k 63/75 ECE 0.064 213 ms
jev (plain wording) AUROC 0.970 [email protected] 0.66 top-k 52/75
nli mnli-base AUROC 0.790 [email protected] 0.28 top-k 38/75 ECE 0.263 1.5 s
nli zeroshot-v2 AUROC 0.782 [email protected] 0.43 top-k 39/75 ECE 0.054 1.2 s
The vendor's calibration claim survives our data; the local cross-encoder
ranks below keyword overlap on either checkpoint or wording and is kept
as the measured negative, not shipped. A local backend would need the
logit-readout route over the fleet's 9B model — a separate spike.
Shadow: one Jev call per phase launch (spawned, 10 s cap, silent without
TYPESAFE_API_KEY) records a skill.triage event; the Skill-Use report
carries triage_p beside each skill's Trigger verdict. It selects nothing.
Co-Authored-By: Claude Opus 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01WZb5A2kfVfjpdwSochkuHz
