docs: addendum 5 — Jev, cm-decide, the eval numbers, shadow triage live

Co-Authored-By: Claude Opus 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01WZb5A2kfVfjpdwSochkuHz

docs/NEXT-SESSION.md

@@ -341,3 +341,72 @@ Open after this pass:
    same generator; the OAP-style signed audit record after that.
 4. Memory retrieval is BM25 by design until the first mission shows it is
    the bottleneck; the measurement is the harness's memory assertion.
+
+## Addendum 5 — 2026-09-21, Jev and the decision tier
+
+**What Jev is.** TypeSafe's "System One" model: state + typed questions in,
+probability distributions out, no text. Choice (option + distribution +
+confidence), Score (expected position over ordered levels), Noul (P(yes)).
+Mechanically almost certainly logits read over caller-defined labels with
+the state's KV prefix shared across questions — hence "adding questions
+barely changes response time" and free output tokens. jev-1.13.0,
+$0.042/M input, 64 K context (32 K state), 1,200 rpm, no fine-tuning, no
+self-host, not trained on customer data. Their own limits: no tools, no
+reasoning, "typed is not correct", not the sole authorization mechanism.
+
+**What we built** (`0a2bd6f`, `37eb6bc`): `crates/cm-decide` — the shapes
+above behind one `Decider` trait, the composition patterns as code
+(confidence gate, composite score, rerank), a Jev HTTP backend and a local
+DeBERTa-v3 MNLI cross-encoder backend (candle; `nli` feature, `metal`/
+`cuda`), and `decide-eval` over `eval/skill-triage.json` — 20 mission
+tasks × 53 skills, 75 positives, hand-labelled.
+
+**Measured** (1,060 pairs):
+
+| backend | AUROC | [email protected] | top-k | Brier | ECE | ms/call |
+|---|---|---|---|---|---|---|
+| lexical overlap (the bar) | 0.851 | 0.47 | 48/75 | 0.066 | 0.095 | 0 |
+| **Jev**, name + when_to_use | **0.989** | **0.84** | **63/75** | **0.025** | 0.064 | 213 |
+| Jev, when_to_use only | 0.970 | 0.66 | 52/75 | 0.054 | 0.120 | 191 |
+| NLI mnli-base (local) | 0.79–0.81 | 0.28–0.31 | 35–38 | 0.13–0.18 | 0.17–0.26 | 900–1600 |
+| NLI zeroshot-v2 (local) | 0.78 | 0.30–0.43 | 35–39 | 0.055 | 0.04–0.05 | 900–1200 |
+
+The vendor's calibration claim survives our data. The local cross-encoder
+ranks **below keyword overlap** on either checkpoint or wording and is
+5–7× slower; kept in the tree as the measured negative, not shipped. A
+local backend would have to be logit read-out over the fleet's 9B model —
+a separate spike, and only worth it if the vendor dependency ever bites.
+Caveat on the numbers: 20 cases; the NLI wordings were tried on the same
+set and still lost; Jev's wording was the first written, not tuned. The
+whole eval cost $0.002.
+
+**Shadow skill triage, live on prod.** One Jev call per phase launch on
+the operator's task text (spawned, 10 s cap, silent without
+`TYPESAFE_API_KEY`) → a `skill.triage` event with per-skill p; the
+Skill-Use report carries `triage_p` beside each Trigger verdict; the
+harness asserts the event on `chain` and `microvm` and prints agreement.
+It selects nothing. First datapoint (`01a0c493`, "create CHAIN.md with one
+line and commit"): Jev's top picks `workspace-repo-commit-protocol` 0.63,
+`small-focused-commits` 0.57 — right; the agent read
+`code-review-checklist` (≈0) and nothing else. That is SRA-Bench's
+"agents load skills at the same rate regardless of need" in one row.
+
+**A defect the run exposed** (`0d46f89`): a phase that supplied its own
+`task` and no `done_when` inherited the RECIPE's condition — for
+`research_and_code`, "an implementation for each INT-XX item in
+IMPLEMENTATION_BRIEF" — and the judge honestly failed CHAIN.md on it. Two
+harness scenarios had already tripped on this (gatepolicy, chain); any
+operator-authored custom phase would too. Now: caller task + no caller
+condition → the recipe's `done_when`/`done_when_check` are not inherited.
+
+**Also:** CI run 6483 failed on a pre-existing race in the no-node gate
+test (empty PATH means no `cat`, so the script exits before reading stdin
+and the test's write gets EPIPE on Linux); tolerated as the no-node path
+working. Key lives in prod `.env` + compose passthrough and the local
+override; never in git, never in a mission container.
+
+**Next for this tier, in order:** (1) accumulate agreement rows from real
+missions before any promotion from shadow to selection; (2) the door
+governor on Jev with the human middle band — stops the GLM spend per door
+action and gives the door a third path; (3) memory rerank; (4) paper
+triage in continuous research.