feat(decide): cm-decide — typed calibrated decisions; Jev + local NLI backends; skill triage in shadow

Browse Source

A third kind of decision-maker between deterministic code and a full LLM
call: Choice / Score / Noul questions answered as probability
distributions with a confidence, behind one Decider trait, with the
composition patterns (confidence gating, composite scoring, rerank) as
code. Two backends: TypeSafe's Jev over HTTP, and a DeBERTa-v3 MNLI
cross-encoder run in-process with candle (feature nli; metal/cuda).

decide-eval measures a backend on labelled cases the way judge-eval
measures the judge. eval/skill-triage.json: 20 mission tasks × 53 skills,
75 positives, hand-labelled. Measured 2026-09-21:

  lexical overlap        AUROC 0.851  [email protected] 0.47  top-k 48/75  ECE 0.095
  jev (named wording)    AUROC 0.989  [email protected] 0.84  top-k 63/75  ECE 0.064  213 ms
  jev (plain wording)    AUROC 0.970  [email protected] 0.66  top-k 52/75
  nli mnli-base          AUROC 0.790  [email protected] 0.28  top-k 38/75  ECE 0.263  1.5 s
  nli zeroshot-v2        AUROC 0.782  [email protected] 0.43  top-k 39/75  ECE 0.054  1.2 s

The vendor's calibration claim survives our data; the local cross-encoder
ranks below keyword overlap on either checkpoint or wording and is kept
as the measured negative, not shipped. A local backend would need the
logit-readout route over the fleet's 9B model — a separate spike.

Shadow: one Jev call per phase launch (spawned, 10 s cap, silent without
TYPESAFE_API_KEY) records a skill.triage event; the Skill-Use report
carries triage_p beside each skill's Trigger verdict. It selects nothing.

Co-Authored-By: Claude Opus 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01WZb5A2kfVfjpdwSochkuHz

...

This commit is contained in:

Omar Sobh

2026-09-21 10:08:31 -05:00

co-authored by Claude Opus 5

parent 650a556029

commit 0a2bd6f868

17 changed files with 3203 additions and 87 deletions

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Download Patch File Download Diff File Expand all files Collapse all files

Cargo.lock

Generated

+1681 -85

File diff suppressed because it is too large Load Diff