0d9498ec6ebda8116a4596877c381d751e08dd8c
4
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Omar SobhandClaude Opus 5
|
c812b714f4 |
fix(evaluator): the verification sandbox never ran a command
`evaluator_tools::Sandbox::run` shelled out to `tokio::process::Command::new
("docker")`. The server image installs `git ca-certificates chromium
fonts-liberation` and nothing else, so in production every verification
command failed to spawn.
The failure was invisible in the worst way. `Sandbox::run` deliberately turns
execution failures into evidence text rather than errors, so a judge reasons
about "that command did not run" instead of the pass collapsing. With no
`docker` binary every command returned COULD NOT RUN, the judge correctly
concluded it could not verify, and fail-closed returned "not met". The
verdicts were right. The verification never happened — and the adversarial
validation that appeared to prove the feature working proved fail-closed
working instead.
The second defect made it worse: `checks` recorded the *attempt*, pushed
before the command ran, so a verdict reached with a dead sandbox reported
"verified by 10 checks" — a stronger claim than "no checks at all", made on
weaker evidence.
- New `container_exec` routes execution through the Docker API via bollard,
which was already a dependency and already reaches the daemon through the
socket proxy. Captures the exit code (absent from the old helper) and keeps
stdout and stderr apart (`LogOutput`'s Display merged them, which is why
nothing downstream could tell JSON from a progress bar). `security_scan`
parses stdout alone; `benchmark_runner` needs both.
- `ExecOutput::success()` requires `Some(0)`. An unreadable status is not
success — `commit_policy = "on_green_tests"` will gate on this, and
"unknown" reading as "green" would push untested work.
- `Sandbox::run` returns a `CheckOutcome` carrying `ran`/`refused`/
`exit_code`. `Verdict::verified_checks()` counts executions, not attempts.
- The UI gains a third state: "could not verify (N attempted, 0 ran)" —
precisely the case that used to render as verified.
- Regression tests reproduce the production shape: two checks recorded,
neither executed, `was_verified() == false`; plus a failing suite (exit 101)
still counting as verification, because that is something the judge learned
rather than was told.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
Omar Sobh
|
d8c8793c4a |
ci fixes: cargo fmt, eslint entities, max-lines split
CI on
|
||
|
Omar Sobh
|
854a617777 |
task #23: retire per-team ZeroClaw container coords (Option A)
Missions never populated teams.zeroclaw_container /
teams.zeroclaw_gateway_url — those were research/loops-era columns
for long-lived per-team containers. Every mission-materialized team
runs inside the SHARED runtime as claws-as-agents provisioned via
RuntimeProvisioner. Reading zeroclaw_container on a mission row
always came up NULL, making security_scan + benchmark_runner
silently fail with "mission has no team container yet."
Changes:
- migrations/0054_drop_teams_zeroclaw_columns.sql — DROP both
columns.
- cm-db/src/repo/teams.rs — delete dead helpers
team_container_coords + set_team_container_coords.
- cm-api/src/security_scan.rs — replace team_container_for_mission
with exec_target(pool, mission_id): container from env
CLAWMATES_RUNTIME_CONTAINER (default clawmates-runtime); workdir
from env CLAWMATES_MISSIONS_ROOT + /{mission_id}/repo
(same convention pdf_renderer uses); precondition that mission
must have repo_id bound.
- cm-api/src/benchmark_runner.rs — same shape.
Follow-up (not in this commit): mission_orchestrator + compose stack
still need to wire a per-mission repo checkout under
CLAWMATES_MISSIONS_ROOT before scan/bench actually produce findings.
Columns cleanup here removes the misleading silent-fail; the
missing-checkout gap is now surfaced with a clear error.
Verified: SQLX_OFFLINE=true cargo check --workspace + cargo test
-p cm-api --test mission_orchestrator both green.
Closes task #23.
|
||
|
Omar SobhandClaude Opus 4.7
|
f843c9ddb1 |
slice 7: before/after benchmark runner
Executes a benchmark harness inside the mission's team container
and records the resulting metrics as a benchmark_snapshots row keyed
on (phase_id, iteration). Baseline pass (iteration=0) captures
before_metrics; each post-iteration call captures after_metrics +
computes delta vs baseline.
Rust surface:
- cm_db::repo::missions::upsert_benchmark_snapshot / benchmark_snapshots_for
- cm_api::benchmark_runner::{baseline, after_iteration, run}
- Harness enum: Auto | Criterion | CargoBench | VitestBench |
PytestBench | Shell (each with a command() vector)
- Auto detection peeks at the repo layout inside the container
(Cargo.toml → CargoBench, package.json → VitestBench, pyproject
→ PytestBench). Falls back to a Shell echo when nothing
identifiable.
- Bencher-format line parser extracts (name, ns_per_iter,
plusminus) so criterion + `cargo bench` output become structured
samples the canvas can diff.
- compute_delta pairs samples by name, emits {before_ns, after_ns,
delta_pct, direction: improved|regressed}.
API:
- POST /api/missions/{id}/benchmark { phase_id, slot, iteration? }
triggers baseline or after run and returns the mission's full
snapshot list.
- GET /api/missions/{id} now includes `benchmarks[]` in the detail
payload.
Frontend:
- New Benchmarks tab on MissionCanvas with iteration + driver
header, plus a 4-column grid (bench / before / after / Δ%) when
delta samples are present. Improved deltas render green,
regressions red.
- TS types + triggerBenchmark() helper in lib/api/missions.ts.
Wiring notes:
- team_container_for_mission reads teams.zeroclaw_container — that's
populated by topology_worker::try_team_gateway_url on first run,
so trigger baseline AFTER the mission's first phase spawns the
container.
- Not auto-fired yet by phase execution; that's the "template phase
executor" work that spans Slices 4-8. Manual API trigger works
today; automated hook is a follow-up.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
|