Omar SobhandClaude Opus 5
0a2bd6f868
A third kind of decision-maker between deterministic code and a full LLM call: Choice / Score / Noul questions answered as probability distributions with a confidence, behind one Decider trait, with the composition patterns (confidence gating, composite scoring, rerank) as code. Two backends: TypeSafe's Jev over HTTP, and a DeBERTa-v3 MNLI cross-encoder run in-process with candle (feature nli; metal/cuda). decide-eval measures a backend on labelled cases the way judge-eval measures the judge. eval/skill-triage.json: 20 mission tasks × 53 skills, 75 positives, hand-labelled. Measured 2026-09-21: lexical overlap AUROC 0.851 [email protected] 0.47 top-k 48/75 ECE 0.095 jev (named wording) AUROC 0.989 [email protected] 0.84 top-k 63/75 ECE 0.064 213 ms jev (plain wording) AUROC 0.970 [email protected] 0.66 top-k 52/75 nli mnli-base AUROC 0.790 [email protected] 0.28 top-k 38/75 ECE 0.263 1.5 s nli zeroshot-v2 AUROC 0.782 [email protected] 0.43 top-k 39/75 ECE 0.054 1.2 s The vendor's calibration claim survives our data; the local cross-encoder ranks below keyword overlap on either checkpoint or wording and is kept as the measured negative, not shipped. A local backend would need the logit-readout route over the fleet's 9B model — a separate spike. Shadow: one Jev call per phase launch (spawned, 10 s cap, silent without TYPESAFE_API_KEY) records a skill.triage event; the Skill-Use report carries triage_p beside each skill's Trigger verdict. It selects nothing. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01WZb5A2kfVfjpdwSochkuHz
37 lines
1.3 KiB
TOML
37 lines
1.3 KiB
TOML
[package]
|
|
name = "cm-decide"
|
|
version = "0.1.0"
|
|
edition.workspace = true
|
|
rust-version.workspace = true
|
|
license.workspace = true
|
|
publish.workspace = true
|
|
description = "Typed, calibrated gut-check decisions (Choice / Score / Noul) behind one trait, with a hosted and a local backend"
|
|
|
|
[features]
|
|
default = ["jev"]
|
|
# TypeSafe's Jev over HTTP.
|
|
jev = ["dep:reqwest"]
|
|
# A local NLI cross-encoder (DeBERTa-v3 MNLI) run in-process with candle.
|
|
nli = ["dep:candle-core", "dep:candle-nn", "dep:candle-transformers", "dep:tokenizers", "dep:hf-hub"]
|
|
metal = ["candle-core/metal", "candle-nn/metal", "candle-transformers/metal"]
|
|
cuda = ["candle-core/cuda", "candle-nn/cuda", "candle-transformers/cuda"]
|
|
|
|
[dependencies]
|
|
async-trait = "0.1"
|
|
serde = { workspace = true }
|
|
serde_json = { workspace = true }
|
|
thiserror = { workspace = true }
|
|
tokio = { workspace = true }
|
|
reqwest = { version = "0.12", default-features = false, features = ["json", "rustls-tls"], optional = true }
|
|
candle-core = { version = "0.11", optional = true }
|
|
candle-nn = { version = "0.11", optional = true }
|
|
candle-transformers = { version = "0.11", optional = true }
|
|
tokenizers = { version = "0.21", default-features = false, features = ["onig"], optional = true }
|
|
hf-hub = { version = "0.4", optional = true }
|
|
|
|
[dev-dependencies]
|
|
tokio = { workspace = true }
|
|
|
|
[lints]
|
|
workspace = true
|