feat(decide): cm-decide — typed calibrated decisions; Jev + local NLI backends; skill triage in shadow
deploy / test (push) Failing after 1m54s
deploy / build (push) Skipped

A third kind of decision-maker between deterministic code and a full LLM
call: Choice / Score / Noul questions answered as probability
distributions with a confidence, behind one Decider trait, with the
composition patterns (confidence gating, composite scoring, rerank) as
code. Two backends: TypeSafe's Jev over HTTP, and a DeBERTa-v3 MNLI
cross-encoder run in-process with candle (feature nli; metal/cuda).

decide-eval measures a backend on labelled cases the way judge-eval
measures the judge. eval/skill-triage.json: 20 mission tasks × 53 skills,
75 positives, hand-labelled. Measured 2026-09-21:

  lexical overlap        AUROC 0.851  [email protected] 0.47  top-k 48/75  ECE 0.095
  jev (named wording)    AUROC 0.989  [email protected] 0.84  top-k 63/75  ECE 0.064  213 ms
  jev (plain wording)    AUROC 0.970  [email protected] 0.66  top-k 52/75
  nli mnli-base          AUROC 0.790  [email protected] 0.28  top-k 38/75  ECE 0.263  1.5 s
  nli zeroshot-v2        AUROC 0.782  [email protected] 0.43  top-k 39/75  ECE 0.054  1.2 s

The vendor's calibration claim survives our data; the local cross-encoder
ranks below keyword overlap on either checkpoint or wording and is kept
as the measured negative, not shipped. A local backend would need the
logit-readout route over the fleet's 9B model — a separate spike.

Shadow: one Jev call per phase launch (spawned, 10 s cap, silent without
TYPESAFE_API_KEY) records a skill.triage event; the Skill-Use report
carries triage_p beside each skill's Trigger verdict. It selects nothing.

Co-Authored-By: Claude Opus 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01WZb5A2kfVfjpdwSochkuHz
This commit is contained in:
Omar Sobh
2026-09-21 10:08:31 -05:00
co-authored by Claude Opus 5
parent 650a556029
commit 0a2bd6f868
17 changed files with 3203 additions and 87 deletions
@@ -130,6 +130,9 @@ services:
CLAWMATES_DOOR_GOVERNOR: "1"
# Read by the `glm` provider's api_key_env in clawmates.toml.
ZAI_API_KEY: ${ZAI_API_KEY:?set in .env}
# TypeSafe Jev — shadow skill triage (cm-api skill_triage.rs). Optional:
# unset, the triage records nothing. Server-side only; never a mission's.
TYPESAFE_API_KEY: ${TYPESAFE_API_KEY:-}
# ElevenLabs GenFM, for rendering the Continuous Research episode.
#
# SERVER-SIDE ONLY, deliberately. It is NOT in