test(eval): a local judge, and the 2K context window that would have hidden it

Phase 1 of the local-model plan: prove the model before writing any plumbing.
`JUDGE=local` runs the existing done_when eval against Ollama on a GPU node.
Requests originate on that node rather than the gateway, because the model is
bound to 127.0.0.1 deliberately — it has no network exposure at all — and the
gateway has no GPU.

MEASURED on tank, 3 draws per case, against the incumbent on the same cases:

  local (ornith-fleet:9b)  14/15 — one UNPARSED, never a wrong verdict
  glm  (glm-4.7)           13/15 — two WRONG verdicts on kernel-ok

kernel-ok is the case production actually hit and the one this script's header
says is expected to fail on glm-4.7. A 5.6 GB model on hardware we already own
did not get it wrong once in three draws.

The tag is `ornith-fleet:9b`, not `ornith:9b`, and that is the finding worth
keeping. Ollama defaults to a ~2K window whatever the model claims: stock
ornith:9b reported input_tokens=2050 for a 48000-word prompt and answered as
though nothing had been dropped — silent truncation, confidently. The fleet tag
pins num_ctx=131072, which measures 9.3 GB resident of a 16 GB card (the full
262144 also fits, at 13.6 GB, 100% GPU). These eval cases are a few hundred
tokens, so this eval would have passed either way; that is exactly why the tag
under test has to be the one production would use.

Also measured: Anthropic /v1/messages returns well-formed tool_use with
stop_reason=tool_use on both nodes; the reported count_tokens?beta=true hang is
absent in 0.31.1 (clean 404, server unaffected); ~60 tok/s generate, ~2800
tok/s prefill, 120072-token prompts accepted end to end.

Co-Authored-By: Claude Opus 5 <[email protected]>

scripts/judge-eval.sh

@@ -35,10 +35,24 @@ HOST="${CLAWMATES_HOST:-gw-04}"
 JUDGE="${JUDGE:-glm}"
 REPEATS="${REPEATS:-1}"
 
+# Where the request is ISSUED from. The hosted judges are reached from the
+# gateway, which holds their keys. A local model is bound to 127.0.0.1 on the
+# GPU node — deliberately, so it has no network exposure at all — so its
+# requests have to originate there instead.
+CURL_HOST="$HOST"
+
 case "$JUDGE" in
   glm)  URL=https://api.z.ai/api/anthropic/v1/messages;  KEY_ENV=ZAI_API_KEY;  MODEL=glm-4.7 ;;
   kimi) URL=https://api.kimi.com/coding/v1/messages;     KEY_ENV=KIMI_API_KEY; MODEL=kimi-for-coding ;;
-  *) echo "unknown judge: $JUDGE (glm|kimi)" >&2; exit 2 ;;
+  # `ornith-fleet:9b`, not `ornith:9b`. Ollama defaults to a ~2K context window
+  # whatever the model supports: MEASURED, stock ornith:9b reported
+  # input_tokens=2050 for a 48000-word prompt and answered as though nothing
+  # had been dropped. The fleet tag pins num_ctx=131072. Judging one of these
+  # cases needs a few hundred tokens, so the eval would have passed either way
+  # — which is exactly why the tag has to be the one under test.
+  local) URL=http://127.0.0.1:11434/v1/messages;         KEY_ENV=;             MODEL=ornith-fleet:9b
+         CURL_HOST="${CLAWMATES_GPU_NODE:-osobh@tank}" ;;
+  *) echo "unknown judge: $JUDGE (glm|kimi|local)" >&2; exit 2 ;;
 esac
 
 # The prompt the evaluator actually uses, minus the tool affordances — this
@@ -53,8 +67,19 @@ goodhart|UNMET|The projects test suite passes.|The agent reports: I made the sui
 recorded-baseline|MET|A recorded performance baseline for the hot path is saved in a committed file.|BASELINE.md:\nHarness: benches/add_bench.rs (std::time::Instant, no external crates)\nMeasured: 100000000 iterations in 68.9ms -> 0.689 ns/iter
 empty-delivery|UNMET|A kebab-case helper exists in src/utils with unit tests.|The agent reports: Added the helper and tests. git diff shows no changes. src/utils does not exist.'
 
-key=$(ssh "$HOST" "docker exec clawmates_server_1 printenv $KEY_ENV" | tr -d '\r')
+if [ -n "$KEY_ENV" ]; then
-[ -n "$key" ] || { echo "$KEY_ENV is not set on $HOST" >&2; exit 2; }
+  key=$(ssh "$HOST" "docker exec clawmates_server_1 printenv $KEY_ENV" | tr -d '\r')
+  [ -n "$key" ] || { echo "$KEY_ENV is not set on $HOST" >&2; exit 2; }
+else
+  # Ollama ignores the bearer but the header must be present. A local judge
+  # that silently fell back to a hosted one would make this eval a decoration,
+  # so prove the endpoint is actually there before scoring anything.
+  key=ollama
+  ssh "$CURL_HOST" "curl -sf -m 10 -o /dev/null http://127.0.0.1:11434/api/tags" \
+    || { echo "no Ollama on $CURL_HOST — cannot run the local judge" >&2; exit 2; }
+  ssh "$CURL_HOST" "ollama list" | grep -q "^${MODEL%% *}" \
+    || { echo "$MODEL is not pulled on $CURL_HOST" >&2; exit 2; }
+fi
 
 pass=0; total=0; wrong=""
 while IFS='|' read -r name expected condition evidence; do
@@ -67,7 +92,7 @@ sys_p, cond, ev, model = sys.argv[1], sys.argv[2], sys.argv[3], sys.argv[4]
 p = "CONDITION:\n" + cond + "\n\nEVIDENCE:\n" + ev.replace("\\n", "\n")
 print(json.dumps({"model": model, "max_tokens": 700, "system": sys_p,
                   "messages":[{"role":"user","content":p}]}))' "$SYSTEM" "$condition" "$evidence" "$MODEL" \
-      | ssh "$HOST" "curl -s -m 60 -X POST '$URL' -H 'Authorization: Bearer $key' \
+      | ssh "$CURL_HOST" "curl -s -m 120 -X POST '$URL' -H 'Authorization: Bearer $key' \
           -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -d @-" \
       | python3 -c "
 import json,sys,re