Durable topology jobs (3+4/4): background worker + async run API
ci / gates (push) Has been cancelled
ci / rust (push) Has been cancelled
ci / sandbox-k8s (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / e2e (push) Has been cancelled

POST /api/topologies/run now ENQUEUES a durable job and returns 202
{run_id, status:queued} instead of executing inside the HTTP request — the
prerequisite for long-horizon runs (no client/proxy/LB timeout, survives
restarts).

topology_worker: a spawned loop that requeues stale running jobs, claims the
next queued one (CAS via FOR UPDATE SKIP LOCKED), drives it through
execute_resumable, and checkpoints RunProgress after every step; on crash the
stale sweep requeues it and the next claim resumes from the last checkpoint.
Wired into server startup beside the scheduler + resume sweeper.

GET /api/topology-runs/{id} now reports lifecycle status/kind/error/checkpoint
+ the result blob (kept the `comparison` field name for back-compat with the
compare UI; null until completed). list_runs includes status + kind.

Tests: durable lifecycle (enqueue→claim→checkpoint→complete) + stale-requeue
resume, both green; p0 endpoints (compare path) unchanged. 13 + 2 tests pass,
clippy clean.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
This commit is contained in:
Omar Sobh
2026-06-17 19:12:54 -07:00
co-authored by Claude Opus 4.8
parent fc9bfc3e61
commit 272669e1f5
5 changed files with 244 additions and 30 deletions
+4
View File
@@ -188,6 +188,10 @@ async fn run() -> Result<(), String> {
// Routine firings (§7.6).
cm_scheduler::Scheduler::new(pool.clone(), runtime.clone())
.spawn(std::time::Duration::from_secs(5));
// Durable topology run jobs: claim queued runs, drive + checkpoint per step,
// resume stale ones after a crash. Long-horizon topologies run here, not in
// the HTTP request.
cm_api::topology_worker::spawn(pool.clone(), std::time::Duration::from_secs(3));
// Hosted identity (Clerk / OIDC): pin the issuer and load its JWKS.
let auth_verifier = match config.auth.mode {