feat(judge): Kimi as a fallback independent judge when GLM cannot answer
deploy / test (push) Successful in 5m32s
deploy / build (push) Successful in 6m4s

GLM's plan limit ran out for the second time in a month, and with one judge
every conditioned phase on every mission fails until it resets. When the
primary independent judge errors, CLAWMATES_VALIDATOR_FALLBACK_MODEL (prod:
kimi:kimi-for-coding) judges instead, through the same independence checks
(one shared function) plus one more: never the primary's own family.

Measured first: judge-eval 15 cases x 3, kimi-for-coding 44/45 vs glm-5.3
43/45; goodhart — the false positive that once ruled Kimi out — 3/3.

When both fail, the primary's error leads so the phase runner still reads the
z.ai plan-limit code and does not spend the pass. Prod's kimi provider moved
to the anthropic format the eval used (host config, backed up).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
Omar Sobh
2026-09-23 07:32:41 -05:00
co-authored by Claude Opus 5.5
parent a13f01e09c
commit abfba832e1
3 changed files with 167 additions and 2 deletions
+5
View File
@@ -47,6 +47,11 @@
# a verification plan before reading the evidence (evaluator.rs), neither of
# which this script exercises.
#
# MEASURED 2026-09-23, kimi-for-coding, three draws of all fifteen: 44/45.
# goodhart 3/3 (its false positive on the original five is gone); the one
# miss is a should-panic-hack draw, the same shape GLM misses. Wired as the
# fallback judge (CLAWMATES_VALIDATOR_FALLBACK_MODEL), not the primary.
#
# Usage:
#
# scripts/judge-eval.sh # the configured validator