feat(judge): room to analyse — and a panic in the evidence path
deploy / test (push) Successful in 4m4s
deploy / build (push) Successful in 5m11s

Three changes, one of them a live bug.

**The bug.** `phase_summarizer` truncated agent output with `&s[..remaining]`,
a BYTE slice of arbitrary UTF-8. Agent turn output routinely carries arrows,
box-drawing and emoji, so a cut landing mid-character panics — taking down the
evaluation sweep for that phase, triggered by nothing more than an agent
writing a long enough line with a non-ASCII character at the wrong offset.
Replaced with `clamp_to_char_boundary`, tested across every cut offset of a
pure-4-byte string.

It is precisely the bug the clawhdf5 agents found and fixed in
`clawhdf5-migrate/src/validate.rs` this week — in our own code, in the path
that feeds the judge.

**Evidence budget** 60 KB -> 120 KB. Output headroom is worthless if the judge
cannot see the work: the verdict is only as good as what reaches it.

**Judge max_tokens** 2048 -> 16384. glm-5.3 is a reasoning model that spends
most of its budget on a `thinking` block before writing the verdict, and
running out mid-thought truncates it. A truncated verdict parses as empty and
FAILS CLOSED, burning one of the phase's passes on a judge that never answered
— how mission 01a00bbb lost one.

Measured ceiling: z.ai accepts max_tokens up to 131072 on both glm-5.1 and
glm-5.3 (131073 -> 400, "限制数值范围[1,131072]"), so 16384 is chosen for cost
and latency rather than capability, and only emitted tokens are billed.

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
Omar Sobh
2026-08-16 21:59:01 -07:00
co-authored by Claude Opus 5
parent 9c4b0722e8
commit 3cc65c22c4
2 changed files with 76 additions and 10 deletions
+60 -2
View File
@@ -27,7 +27,29 @@ const POLL_INTERVAL: Duration = Duration::from_secs(30);
/// Cap the raw material we send to the model. Missions can produce
/// hundreds of KB of agent output; we slice by turn and by phase
/// artifact but still bound the total prompt.
const MAX_OUTPUT_BYTES: usize = 60_000;
const MAX_OUTPUT_BYTES: usize = 120_000;
/// The longest prefix of `s` that is at most `max_bytes` and ends on a
/// character boundary.
///
/// `&s[..max_bytes]` PANICS when the cut lands inside a multi-byte character,
/// and `s` here is agent-authored turn output — arbitrary UTF-8, routinely
/// containing arrows, box-drawing and emoji. The panic would take down the
/// evaluation sweep for a phase whose only crime was writing a long enough
/// line with a non-ASCII character at the wrong offset.
///
/// Exactly the bug the clawhdf5 agents found and fixed in
/// `clawhdf5-migrate/src/validate.rs` this week, in our own code.
fn clamp_to_char_boundary(s: &str, max_bytes: usize) -> &str {
if s.len() <= max_bytes {
return s;
}
let mut end = max_bytes;
while end > 0 && !s.is_char_boundary(end) {
end -= 1;
}
&s[..end]
}
fn model_name() -> String {
std::env::var("CLAWMATES_SUMMARIZER_MODEL").unwrap_or_else(|_| DEFAULT_MODEL.to_string())
@@ -254,7 +276,7 @@ async fn collect_material(
concat.push_str(&format!("\n\n── turn {} ──\n", i + 1));
let remaining = MAX_OUTPUT_BYTES.saturating_sub(concat.len());
if s.len() > remaining {
concat.push_str(&s[..remaining]);
concat.push_str(clamp_to_char_boundary(&s, remaining));
concat.push_str("\n… (truncated)");
} else {
concat.push_str(&s);
@@ -578,3 +600,39 @@ async fn record_error(
.map_err(|e| format!("record error: {e}"))?;
Ok(())
}
#[cfg(test)]
mod tests {
use super::*;
/// Agent output is arbitrary UTF-8. A byte-offset cut that lands inside a
/// multi-byte character must not panic — that panic would take down the
/// evaluation sweep for the phase, and the only trigger is an agent
/// happening to write a long enough line containing a non-ASCII character.
#[test]
fn truncation_never_splits_a_multibyte_character() {
// 4-byte characters, so every offset not a multiple of 4 is
// mid-character and would panic a naive `&s[..cut]`.
let s = "😀".repeat(10);
for cut in 0..=s.len() {
let out = clamp_to_char_boundary(&s, cut);
assert!(out.len() <= cut, "must respect the budget at cut={cut}");
assert!(s.starts_with(out), "must stay a prefix at cut={cut}");
}
}
/// Mixed-width text: the cut must land on a boundary, never inside `é`.
#[test]
fn truncation_handles_mixed_width_text() {
let s = "héllo wörld";
for cut in 0..=s.len() {
let out = clamp_to_char_boundary(s, cut);
assert!(s.starts_with(out));
}
}
#[test]
fn truncation_returns_everything_when_it_fits() {
assert_eq!(clamp_to_char_boundary("héllo", 100), "héllo");
}
}