From 4e342d1ff781a18c5aa4fbe14e098450a026751d Mon Sep 17 00:00:00 2001 From: Omar Sobh Date: Fri, 18 Sep 2026 22:34:05 -0500 Subject: [PATCH] perf(judge): a command may return 64 KB, so a deliverable is read once MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 7 of 9 verdicts ran to the 12-check cap. The checks say why: on a research mission, 8 of 12 commands read research/REPORT.md — cat, then head -119, tail -120, sed -n 80,200p and three greps. `cat` had come back truncated at 11,983 bytes because the per-command cap was 12 KB and the report was ~18 KB, so the judge reassembled the file in slices. Five extra rounds, each resending the whole conversation, to read one deliverable. The microVM verdicts used 5–7 checks because MICROVM.md is two lines. The cap was sized for test-suite output and applied to deliverables. 64 KB now. With compact_earlier_results shrinking a result to 800 bytes after its round, one 64 KB read costs one round; the slicing it replaces cost five. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01WZb5A2kfVfjpdwSochkuHz --- crates/cm-api/src/evaluator_tools.rs | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/crates/cm-api/src/evaluator_tools.rs b/crates/cm-api/src/evaluator_tools.rs index aedfc4f..9790d96 100644 --- a/crates/cm-api/src/evaluator_tools.rs +++ b/crates/cm-api/src/evaluator_tools.rs @@ -50,7 +50,16 @@ const COMMAND_TIMEOUT: Duration = Duration::from_secs(180); /// Cap on what one command may return to the model. Test suites are chatty and /// the judge pays for every byte; the tail is where failures live, so when /// output overflows we keep both ends and drop the middle. -const MAX_OUTPUT_BYTES: usize = 12_000; +/// +/// 64 KB, up from 12 KB on 2026-09-18. The smaller cap was sized for test +/// output and applied to deliverables: a research REPORT.md of ~18 KB came +/// back truncated from `cat`, and the judge — correctly — reassembled it with +/// `head -119`, `tail -120`, `sed -n 80,200p` and three greps, five extra +/// rounds each resending the whole conversation. 7 of 9 verdicts ran to the +/// 12-check cap that way. Since `compact_earlier_results` shrinks a result to +/// 800 bytes once its round is over, one 64 KB read costs one round; the +/// slicing it replaces cost five. +const MAX_OUTPUT_BYTES: usize = 64_000; /// Programs the judge may run. Every one either reports state or runs a /// project's own checks — none of them edit the tree.