From 2ebba77f7f32b8e30482de8312aadc0e09adbb99 Mon Sep 17 00:00:00 2001 From: Omar Sobh Date: Mon, 21 Sep 2026 10:32:11 -0500 Subject: [PATCH] =?UTF-8?q?docs:=20addendum=205=20=E2=80=94=20Jev,=20cm-de?= =?UTF-8?q?cide,=20the=20eval=20numbers,=20shadow=20triage=20live?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01WZb5A2kfVfjpdwSochkuHz --- docs/NEXT-SESSION.md | 69 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 69 insertions(+) diff --git a/docs/NEXT-SESSION.md b/docs/NEXT-SESSION.md index 1383f7d..d50a545 100644 --- a/docs/NEXT-SESSION.md +++ b/docs/NEXT-SESSION.md @@ -341,3 +341,72 @@ Open after this pass: same generator; the OAP-style signed audit record after that. 4. Memory retrieval is BM25 by design until the first mission shows it is the bottleneck; the measurement is the harness's memory assertion. + +## Addendum 5 — 2026-09-21, Jev and the decision tier + +**What Jev is.** TypeSafe's "System One" model: state + typed questions in, +probability distributions out, no text. Choice (option + distribution + +confidence), Score (expected position over ordered levels), Noul (P(yes)). +Mechanically almost certainly logits read over caller-defined labels with +the state's KV prefix shared across questions — hence "adding questions +barely changes response time" and free output tokens. jev-1.13.0, +$0.042/M input, 64 K context (32 K state), 1,200 rpm, no fine-tuning, no +self-host, not trained on customer data. Their own limits: no tools, no +reasoning, "typed is not correct", not the sole authorization mechanism. + +**What we built** (`0a2bd6f`, `37eb6bc`): `crates/cm-decide` — the shapes +above behind one `Decider` trait, the composition patterns as code +(confidence gate, composite score, rerank), a Jev HTTP backend and a local +DeBERTa-v3 MNLI cross-encoder backend (candle; `nli` feature, `metal`/ +`cuda`), and `decide-eval` over `eval/skill-triage.json` — 20 mission +tasks × 53 skills, 75 positives, hand-labelled. + +**Measured** (1,060 pairs): + +| backend | AUROC | F1@0.5 | top-k | Brier | ECE | ms/call | +|---|---|---|---|---|---|---| +| lexical overlap (the bar) | 0.851 | 0.47 | 48/75 | 0.066 | 0.095 | 0 | +| **Jev**, name + when_to_use | **0.989** | **0.84** | **63/75** | **0.025** | 0.064 | 213 | +| Jev, when_to_use only | 0.970 | 0.66 | 52/75 | 0.054 | 0.120 | 191 | +| NLI mnli-base (local) | 0.79–0.81 | 0.28–0.31 | 35–38 | 0.13–0.18 | 0.17–0.26 | 900–1600 | +| NLI zeroshot-v2 (local) | 0.78 | 0.30–0.43 | 35–39 | 0.055 | 0.04–0.05 | 900–1200 | + +The vendor's calibration claim survives our data. The local cross-encoder +ranks **below keyword overlap** on either checkpoint or wording and is +5–7× slower; kept in the tree as the measured negative, not shipped. A +local backend would have to be logit read-out over the fleet's 9B model — +a separate spike, and only worth it if the vendor dependency ever bites. +Caveat on the numbers: 20 cases; the NLI wordings were tried on the same +set and still lost; Jev's wording was the first written, not tuned. The +whole eval cost $0.002. + +**Shadow skill triage, live on prod.** One Jev call per phase launch on +the operator's task text (spawned, 10 s cap, silent without +`TYPESAFE_API_KEY`) → a `skill.triage` event with per-skill p; the +Skill-Use report carries `triage_p` beside each Trigger verdict; the +harness asserts the event on `chain` and `microvm` and prints agreement. +It selects nothing. First datapoint (`01a0c493`, "create CHAIN.md with one +line and commit"): Jev's top picks `workspace-repo-commit-protocol` 0.63, +`small-focused-commits` 0.57 — right; the agent read +`code-review-checklist` (≈0) and nothing else. That is SRA-Bench's +"agents load skills at the same rate regardless of need" in one row. + +**A defect the run exposed** (`0d46f89`): a phase that supplied its own +`task` and no `done_when` inherited the RECIPE's condition — for +`research_and_code`, "an implementation for each INT-XX item in +IMPLEMENTATION_BRIEF" — and the judge honestly failed CHAIN.md on it. Two +harness scenarios had already tripped on this (gatepolicy, chain); any +operator-authored custom phase would too. Now: caller task + no caller +condition → the recipe's `done_when`/`done_when_check` are not inherited. + +**Also:** CI run 6483 failed on a pre-existing race in the no-node gate +test (empty PATH means no `cat`, so the script exits before reading stdin +and the test's write gets EPIPE on Linux); tolerated as the no-node path +working. Key lives in prod `.env` + compose passthrough and the local +override; never in git, never in a mission container. + +**Next for this tier, in order:** (1) accumulate agreement rows from real +missions before any promotion from shadow to selection; (2) the door +governor on Jev with the human middle band — stops the GLM spend per door +action and gives the door a third path; (3) memory rerank; (4) paper +triage in continuous research.