feat(judge): an independent GLM judge, on the newest model z.ai publishes
deploy / test (push) Successful in 3m58s
deploy / build (push) Successful in 5m29s

Every phase verdict this session was Anthropic grading Anthropic, and the boot
log said so on each start:

    validator_preflight: no CLAWMATES_VALIDATOR_MODEL — phase verdicts are
    judged by the house model, which is NOT an independent check

`evaluator.rs` already preferred a cross-provider judge and refused to call a
same-family one `independent`; the local stack simply had no non-Anthropic
credential. It now carries the same `glm` provider gw-04 has had all along —
`format = "anthropic"` is load-bearing, since z.ai's OpenAI-compatible endpoint
is ToS-throttled for raw SDK access while its Anthropic-format one is not.

Model: glm-5.3, the newest z.ai lists (4.5, 4.5-air, 4.6, 4.7, 5, 5-turbo,
5.1, 5.2, 5.3 as of 2026-08-17). gw-04 still runs glm-4.7.

glm-5.3 is a REASONING model: it emits a `thinking` block before its JSON. Our
SSE parser ignores `thinking_delta` and keeps the text, so the wire shape is
compatible — but on a realistic phase-evidence prompt it spent 819 of the
evaluator's 1024 output tokens. A longer phase would truncate the verdict, and
a truncated verdict parses as empty and FAILS CLOSED, burning one of the
phase's passes on a judge that never answered — precisely how mission 01a00bbb
lost a pass. max_tokens raised to 2048.

Measured before wiring: asked to judge 25 commits claiming INT-01..INT-25 with
tests passing, glm-5.3 returned met=false because the evidence never
established what the brief actually required. That skepticism is the point of
an independent judge.

Boot now reports: `validator_preflight: independent validator glm:glm-5.3
answered`.

The key lives in .env (gitignored), never in this file.

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
Omar Sobh
2026-08-16 21:41:45 -07:00
co-authored by Claude Opus 5
parent b5032a732a
commit 9c4b0722e8
2 changed files with 25 additions and 1 deletions
+8 -1
View File
@@ -625,7 +625,14 @@ async fn judge_with_tools(
model: model.to_string(), model: model.to_string(),
messages: messages.clone(), messages: messages.clone(),
tools: tools.clone(), tools: tools.clone(),
max_tokens: 1024, // Headroom for a REASONING judge. glm-5.3 emits a `thinking`
// block before its JSON and spent 819 of 1024 tokens on a
// realistic phase-evidence prompt; a longer phase would truncate
// the verdict, and a truncated verdict parses as empty and fails
// closed — burning one of the phase's passes on a judge that
// never answered, exactly as the subscription judge did on
// mission 01a00bbb.
max_tokens: 2048,
web_search: false, web_search: false,
}; };
let mut stream = provider.stream(request).await.map_err(|e| e.to_string())?; let mut stream = provider.stream(request).await.map_err(|e| e.to_string())?;
+17
View File
@@ -14,6 +14,23 @@ provider = "openai_compat"
base_url = "http://local-llm:8000/v1" base_url = "http://local-llm:8000/v1"
model = "qwen2.5-72b-instruct" model = "qwen2.5-72b-instruct"
# GLM, so a phase verdict is judged by a DIFFERENT provider family than the one
# that did the work. `evaluator.rs` refuses to call a same-family judge
# `independent`, and until now this stack had only Anthropic credentials — so
# every verdict was Anthropic grading Anthropic and the boot log said so:
#
# validator_preflight: no CLAWMATES_VALIDATOR_MODEL — phase verdicts are
# judged by the house model, which is NOT an independent check
#
# `format = "anthropic"` is required, not cosmetic: z.ai's OpenAI-compatible
# endpoint is ToS-throttled for raw SDK access, while its Anthropic-format
# endpoint accepts them. Mirrors gw-04's clawmates.toml exactly.
[[llm.providers]]
name = "glm"
base_url = "https://api.z.ai/api/anthropic"
api_key_env = "ZAI_API_KEY"
format = "anthropic"
[auth] [auth]
mode = "local" mode = "local"