fix(evaluator): a bare validator model name claimed independence it never had

`CLAWMATES_VALIDATOR_MODEL=gemini-2.5-flash` (or any bare model name) produced
an Anthropic judge grading Anthropic work, recorded `independent = true`.

The chain:

  - `provider_family` reads the SPEC. A bare `gemini-2.5-flash` matches none of
    the known needles, so it returns "unknown" — deliberately NOT "anthropic",
    so it passes the `family == IMPLEMENTER_FAMILY` guard.
  - `Runtime::resolve_provider` (runtime.rs:224) falls back to the DEFAULT
    provider for any spec it cannot route. A bare name has no `provider:` to
    route on, so it silently returns the house Anthropic provider.
  - The existing "no provider registered" guard checks `model.contains(':')`.
    That works for `glm:glm-4.7` — an unrouted colon-spec comes back carrying
    its colon — and can NEVER fire for a bare name.

So the one guarantee this path exists to make (the judge is not the implementer)
was reported as satisfied while being violated. That is the same shape as the
Goodhart incident the independent judge was built after: not a wrong answer, a
wrongly-trusted one.

A validator spec must now name its provider. `names_a_provider` is a named
predicate rather than an inline `contains(':')` so the rule is testable and the
reasoning has somewhere to live.

Found while auditing my own Gemini removal — which turned out to be
behaviour-neutral here (a gemini spec went from family "gemini" to "unknown",
both non-anthropic, same verdict). The bug is pre-existing and independent of
it; removing Gemini only made the bare `gemini-*` spelling more likely to be
left behind in someone's env.

Live config is `glm:glm-4.7`, a proper registry spec, so production behaviour is
unchanged. Negative control: make `names_a_provider` return true unconditionally
and `a_validator_spec_must_name_its_provider` fails.

241 lib tests pass.
This commit is contained in:
Omar Sobh
2026-08-07 14:34:28 -07:00
parent 87f188ae73
commit a20702d55b
+52
View File
@@ -251,6 +251,18 @@ pub fn provider_family(spec: &str) -> String {
"unknown".into() "unknown".into()
} }
/// Does this validator spec name the provider it wants, rather than only a model?
///
/// `Runtime::resolve_provider` routes `provider:model` and falls back to the
/// DEFAULT provider for everything else. That fallback is what makes a bare name
/// dangerous here: it silently yields the house provider, which the independence
/// check then fails to recognise as the house provider — because
/// `provider_family` reads the SPEC, and a bare `gemini-2.5-flash` reads as
/// "unknown", not "anthropic".
fn names_a_provider(spec: &str) -> bool {
spec.contains(':')
}
/// The provider family the mission's agent ran on. /// The provider family the mission's agent ran on.
/// ///
/// Today every mission backend is Claude Code (`agent-claude`), including the /// Today every mission backend is Claude Code (`agent-claude`), including the
@@ -319,6 +331,25 @@ async fn cross_provider_judge(
); );
return None; return None;
} }
// A validator spec MUST name its provider. `resolve_provider` falls back to
// the DEFAULT provider for anything it cannot route (runtime.rs), and for a
// bare model name that fallback is silent: `gemini-2.5-flash` has no colon,
// so it resolved to the house Anthropic provider while `provider_family`
// reported "unknown" — not "anthropic" — and the verdict was recorded
// `independent = true`. An Anthropic judge grading Anthropic work, labelled
// independent, which is the one claim this whole path exists to make honestly.
//
// The colon check below caught the same fallback for `glm:glm-4.7` when the
// `glm` provider was missing, because the unrouted spec comes back WITH its
// colon. It could never catch a bare name.
if !names_a_provider(spec) {
eprintln!(
"evaluator: CLAWMATES_VALIDATOR_MODEL={spec} is not a registry spec \
(expected `provider:model`, e.g. `glm:glm-4.7`) — refusing to judge with \
the default provider and call it independent"
);
return None;
}
let (provider, model) = runtime.resolve_provider(spec); let (provider, model) = runtime.resolve_provider(spec);
if model.contains(':') { if model.contains(':') {
eprintln!( eprintln!(
@@ -784,6 +815,27 @@ mod cross_provider_tests {
/// A family, not a model. Two Claude models share a lineage and most of their /// A family, not a model. Two Claude models share a lineage and most of their
/// failure modes, so `opus` judging `sonnet` is not an independent check. /// failure modes, so `opus` judging `sonnet` is not an independent check.
#[test] #[test]
/// A bare model name must never be accepted as a validator spec.
///
/// `resolve_provider` falls back to the DEFAULT provider for anything it
/// cannot route, and for a bare name that fallback is invisible: the spec
/// has no `provider:` prefix to come back with, so the existing
/// "no provider registered" check cannot see it. The result was an
/// Anthropic judge grading Anthropic work with `independent = true`.
#[test]
fn a_validator_spec_must_name_its_provider() {
for good in ["glm:glm-4.7", "kimi:kimi-for-coding", "runtime:some-alias"] {
assert!(names_a_provider(good), "{good} is a registry spec");
}
// These are the dangerous ones: they resolve to the DEFAULT provider.
for bare in ["gemini-2.5-flash", "claude-sonnet-5", "glm-4.7", ""] {
assert!(
!names_a_provider(bare),
"{bare:?} names no provider and must be refused"
);
}
}
fn every_anthropic_spelling_is_one_family() { fn every_anthropic_spelling_is_one_family() {
for spec in [ for spec in [
"claude-opus-4-8", "claude-opus-4-8",