feat(judge): room to analyse — and a panic in the evidence path
Three changes, one of them a live bug. **The bug.** `phase_summarizer` truncated agent output with `&s[..remaining]`, a BYTE slice of arbitrary UTF-8. Agent turn output routinely carries arrows, box-drawing and emoji, so a cut landing mid-character panics — taking down the evaluation sweep for that phase, triggered by nothing more than an agent writing a long enough line with a non-ASCII character at the wrong offset. Replaced with `clamp_to_char_boundary`, tested across every cut offset of a pure-4-byte string. It is precisely the bug the clawhdf5 agents found and fixed in `clawhdf5-migrate/src/validate.rs` this week — in our own code, in the path that feeds the judge. **Evidence budget** 60 KB -> 120 KB. Output headroom is worthless if the judge cannot see the work: the verdict is only as good as what reaches it. **Judge max_tokens** 2048 -> 16384. glm-5.3 is a reasoning model that spends most of its budget on a `thinking` block before writing the verdict, and running out mid-thought truncates it. A truncated verdict parses as empty and FAILS CLOSED, burning one of the phase's passes on a judge that never answered — how mission 01a00bbb lost one. Measured ceiling: z.ai accepts max_tokens up to 131072 on both glm-5.1 and glm-5.3 (131073 -> 400, "限制数值范围[1,131072]"), so 16384 is chosen for cost and latency rather than capability, and only emitted tokens are billed. Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
co-authored by
Claude Opus 5
parent
9c4b0722e8
commit
3cc65c22c4
@@ -27,7 +27,29 @@ const POLL_INTERVAL: Duration = Duration::from_secs(30);
|
||||
/// Cap the raw material we send to the model. Missions can produce
|
||||
/// hundreds of KB of agent output; we slice by turn and by phase
|
||||
/// artifact but still bound the total prompt.
|
||||
const MAX_OUTPUT_BYTES: usize = 60_000;
|
||||
const MAX_OUTPUT_BYTES: usize = 120_000;
|
||||
|
||||
/// The longest prefix of `s` that is at most `max_bytes` and ends on a
|
||||
/// character boundary.
|
||||
///
|
||||
/// `&s[..max_bytes]` PANICS when the cut lands inside a multi-byte character,
|
||||
/// and `s` here is agent-authored turn output — arbitrary UTF-8, routinely
|
||||
/// containing arrows, box-drawing and emoji. The panic would take down the
|
||||
/// evaluation sweep for a phase whose only crime was writing a long enough
|
||||
/// line with a non-ASCII character at the wrong offset.
|
||||
///
|
||||
/// Exactly the bug the clawhdf5 agents found and fixed in
|
||||
/// `clawhdf5-migrate/src/validate.rs` this week, in our own code.
|
||||
fn clamp_to_char_boundary(s: &str, max_bytes: usize) -> &str {
|
||||
if s.len() <= max_bytes {
|
||||
return s;
|
||||
}
|
||||
let mut end = max_bytes;
|
||||
while end > 0 && !s.is_char_boundary(end) {
|
||||
end -= 1;
|
||||
}
|
||||
&s[..end]
|
||||
}
|
||||
|
||||
fn model_name() -> String {
|
||||
std::env::var("CLAWMATES_SUMMARIZER_MODEL").unwrap_or_else(|_| DEFAULT_MODEL.to_string())
|
||||
@@ -254,7 +276,7 @@ async fn collect_material(
|
||||
concat.push_str(&format!("\n\n── turn {} ──\n", i + 1));
|
||||
let remaining = MAX_OUTPUT_BYTES.saturating_sub(concat.len());
|
||||
if s.len() > remaining {
|
||||
concat.push_str(&s[..remaining]);
|
||||
concat.push_str(clamp_to_char_boundary(&s, remaining));
|
||||
concat.push_str("\n… (truncated)");
|
||||
} else {
|
||||
concat.push_str(&s);
|
||||
@@ -578,3 +600,39 @@ async fn record_error(
|
||||
.map_err(|e| format!("record error: {e}"))?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
/// Agent output is arbitrary UTF-8. A byte-offset cut that lands inside a
|
||||
/// multi-byte character must not panic — that panic would take down the
|
||||
/// evaluation sweep for the phase, and the only trigger is an agent
|
||||
/// happening to write a long enough line containing a non-ASCII character.
|
||||
#[test]
|
||||
fn truncation_never_splits_a_multibyte_character() {
|
||||
// 4-byte characters, so every offset not a multiple of 4 is
|
||||
// mid-character and would panic a naive `&s[..cut]`.
|
||||
let s = "😀".repeat(10);
|
||||
for cut in 0..=s.len() {
|
||||
let out = clamp_to_char_boundary(&s, cut);
|
||||
assert!(out.len() <= cut, "must respect the budget at cut={cut}");
|
||||
assert!(s.starts_with(out), "must stay a prefix at cut={cut}");
|
||||
}
|
||||
}
|
||||
|
||||
/// Mixed-width text: the cut must land on a boundary, never inside `é`.
|
||||
#[test]
|
||||
fn truncation_handles_mixed_width_text() {
|
||||
let s = "héllo wörld";
|
||||
for cut in 0..=s.len() {
|
||||
let out = clamp_to_char_boundary(s, cut);
|
||||
assert!(s.starts_with(out));
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn truncation_returns_everything_when_it_fits() {
|
||||
assert_eq!(clamp_to_char_boundary("héllo", 100), "héllo");
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user