slice 7: before/after benchmark runner
Executes a benchmark harness inside the mission's team container
and records the resulting metrics as a benchmark_snapshots row keyed
on (phase_id, iteration). Baseline pass (iteration=0) captures
before_metrics; each post-iteration call captures after_metrics +
computes delta vs baseline.
Rust surface:
- cm_db::repo::missions::upsert_benchmark_snapshot / benchmark_snapshots_for
- cm_api::benchmark_runner::{baseline, after_iteration, run}
- Harness enum: Auto | Criterion | CargoBench | VitestBench |
PytestBench | Shell (each with a command() vector)
- Auto detection peeks at the repo layout inside the container
(Cargo.toml → CargoBench, package.json → VitestBench, pyproject
→ PytestBench). Falls back to a Shell echo when nothing
identifiable.
- Bencher-format line parser extracts (name, ns_per_iter,
plusminus) so criterion + `cargo bench` output become structured
samples the canvas can diff.
- compute_delta pairs samples by name, emits {before_ns, after_ns,
delta_pct, direction: improved|regressed}.
API:
- POST /api/missions/{id}/benchmark { phase_id, slot, iteration? }
triggers baseline or after run and returns the mission's full
snapshot list.
- GET /api/missions/{id} now includes `benchmarks[]` in the detail
payload.
Frontend:
- New Benchmarks tab on MissionCanvas with iteration + driver
header, plus a 4-column grid (bench / before / after / Δ%) when
delta samples are present. Improved deltas render green,
regressions red.
- TS types + triggerBenchmark() helper in lib/api/missions.ts.
Wiring notes:
- team_container_for_mission reads teams.zeroclaw_container — that's
populated by topology_worker::try_team_gateway_url on first run,
so trigger baseline AFTER the mission's first phase spawns the
container.
- Not auto-fired yet by phase execution; that's the "template phase
executor" work that spans Slices 4-8. Manual API trigger works
today; automated hook is a follow-up.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
This commit is contained in:
co-authored by
Claude Opus 4.7
parent
3ac3d53da7
commit
f843c9ddb1
@@ -10,7 +10,8 @@ use axum::{
|
||||
Json,
|
||||
};
|
||||
use cm_db::repo::missions::{
|
||||
Mission, MissionArtifact, MissionPhase, MissionTask, NewMission, NewMissionPhase,
|
||||
BenchmarkSnapshot, Mission, MissionArtifact, MissionPhase, MissionTask, NewMission,
|
||||
NewMissionPhase,
|
||||
};
|
||||
use serde::{Deserialize, Serialize};
|
||||
use serde_json::Value;
|
||||
@@ -71,6 +72,17 @@ pub struct MissionDetail {
|
||||
pub phases: Vec<MissionPhase>,
|
||||
pub tasks: Vec<MissionTask>,
|
||||
pub artifacts: Vec<MissionArtifact>,
|
||||
pub benchmarks: Vec<BenchmarkSnapshot>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Deserialize)]
|
||||
pub struct BenchmarkTriggerRequest {
|
||||
pub phase_id: Uuid,
|
||||
/// Slot: "baseline" (records iteration 0) or "after"
|
||||
/// (records iteration N + delta vs baseline).
|
||||
pub slot: String,
|
||||
#[serde(default)]
|
||||
pub iteration: Option<i32>,
|
||||
}
|
||||
|
||||
// ── Handlers ─────────────────────────────────────────────────────
|
||||
@@ -135,14 +147,46 @@ pub async fn get(
|
||||
let phases = cm_db::repo::missions::phases_for(&state.pool, id).await?;
|
||||
let tasks = cm_db::repo::missions::tasks_for(&state.pool, id).await?;
|
||||
let artifacts = cm_db::repo::missions::artifacts_for(&state.pool, id).await?;
|
||||
let benchmarks = cm_db::repo::missions::benchmark_snapshots_for(&state.pool, id).await?;
|
||||
Ok(Json(MissionDetail {
|
||||
mission,
|
||||
phases,
|
||||
tasks,
|
||||
artifacts,
|
||||
benchmarks,
|
||||
}))
|
||||
}
|
||||
|
||||
/// POST /api/missions/{id}/benchmark — run the benchmark harness
|
||||
/// against a phase. Slot='baseline' records iteration 0's
|
||||
/// before_metrics; slot='after' with iteration=N records the
|
||||
/// after_metrics + computes delta against baseline.
|
||||
pub async fn trigger_benchmark(
|
||||
State(state): State<AppState>,
|
||||
Authed(user): Authed,
|
||||
Path(id): Path<Uuid>,
|
||||
Json(body): Json<BenchmarkTriggerRequest>,
|
||||
) -> Result<Json<Vec<BenchmarkSnapshot>>, ApiError> {
|
||||
// Workspace scope check on the mission — 404 if not visible.
|
||||
let _ = cm_db::repo::missions::get(&state.pool, id, user.workspace_id.as_uuid())
|
||||
.await?
|
||||
.ok_or(ApiError::NotFound)?;
|
||||
let result = match body.slot.as_str() {
|
||||
"baseline" => crate::benchmark_runner::baseline(&state.pool, id, body.phase_id).await,
|
||||
"after" => {
|
||||
let iter = body.iteration.unwrap_or(1);
|
||||
crate::benchmark_runner::after_iteration(&state.pool, id, body.phase_id, iter).await
|
||||
}
|
||||
_ => return Err(ApiError::BadRequest),
|
||||
};
|
||||
if let Err(e) = result {
|
||||
eprintln!("benchmark trigger for mission {id}: {e}");
|
||||
return Err(ApiError::Internal);
|
||||
}
|
||||
let snaps = cm_db::repo::missions::benchmark_snapshots_for(&state.pool, id).await?;
|
||||
Ok(Json(snaps))
|
||||
}
|
||||
|
||||
pub async fn set_status(
|
||||
State(state): State<AppState>,
|
||||
Authed(user): Authed,
|
||||
|
||||
Reference in New Issue
Block a user