Omar SobhandClaude Opus 5
0a2bd6f868
A third kind of decision-maker between deterministic code and a full LLM call: Choice / Score / Noul questions answered as probability distributions with a confidence, behind one Decider trait, with the composition patterns (confidence gating, composite scoring, rerank) as code. Two backends: TypeSafe's Jev over HTTP, and a DeBERTa-v3 MNLI cross-encoder run in-process with candle (feature nli; metal/cuda). decide-eval measures a backend on labelled cases the way judge-eval measures the judge. eval/skill-triage.json: 20 mission tasks × 53 skills, 75 positives, hand-labelled. Measured 2026-09-21: lexical overlap AUROC 0.851 [email protected] 0.47 top-k 48/75 ECE 0.095 jev (named wording) AUROC 0.989 [email protected] 0.84 top-k 63/75 ECE 0.064 213 ms jev (plain wording) AUROC 0.970 [email protected] 0.66 top-k 52/75 nli mnli-base AUROC 0.790 [email protected] 0.28 top-k 38/75 ECE 0.263 1.5 s nli zeroshot-v2 AUROC 0.782 [email protected] 0.43 top-k 39/75 ECE 0.054 1.2 s The vendor's calibration claim survives our data; the local cross-encoder ranks below keyword overlap on either checkpoint or wording and is kept as the measured negative, not shipped. A local backend would need the logit-readout route over the fleet's 9B model — a separate spike. Shadow: one Jev call per phase launch (spawned, 10 s cap, silent without TYPESAFE_API_KEY) records a skill.triage event; the Skill-Use report carries triage_p beside each skill's Trigger verdict. It selects nothing. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01WZb5A2kfVfjpdwSochkuHz
35 lines
1.1 KiB
Plaintext
35 lines
1.1 KiB
Plaintext
/target
|
|
**/node_modules
|
|
frontend/.next
|
|
frontend/out
|
|
frontend/coverage
|
|
frontend/playwright-report
|
|
frontend/test-results
|
|
.env
|
|
.env.local
|
|
*.log
|
|
.DS_Store
|
|
data/
|
|
token.key
|
|
|
|
# Hosted node-agent binaries (built + baked into the frontend image, not committed)
|
|
frontend/public/dl/
|
|
|
|
# Local env backups. `.env` is already ignored above, but a timestamped or
|
|
# suffixed copy of it is not — and these hold real credentials (subscription
|
|
# OAuth token, forge PAT, DB password). Ignore every variant, not just the
|
|
# exact name.
|
|
.env.bak*
|
|
*.env.bak*
|
|
deploy/compose/.env.*
|
|
|
|
# Local-only compose override. NOT for prod or the air-gapped install: it
|
|
# rebinds published ports to loopback, enables the login bypass, and points the
|
|
# runtime at MacBook-specific paths. docker-compose picks this file up
|
|
# automatically, so committing it would silently reconfigure anyone who runs
|
|
# deploy/compose.
|
|
# deploy/compose/docker-compose.override.yml is TRACKED as of 2026-09-18: it
|
|
# holds the fixes for the five local bring-up gaps and every credential in it is
|
|
# a ${VAR:?} reference into .env. It lived only on one laptop until then.
|
|
crates/cm-decide/eval/out/
|