fix(runs): the composed worker's checkpoint wiped the live log on every node

Composed missions streamed ZERO bytes while solo missions streamed fine. Same
executor, same command, same guest — `HubVms::run` is a straight passthrough —
and the node logged a tail starting for all five graph nodes against the correct
outer run id, with no errors. The bytes simply were not there at the end.

Two writers, one column. `fleet.rs` appends live output under `checkpoint.log`;
`topology_runs::checkpoint` wrote `SET checkpoint = $2`, replacing the whole
object. A composed run checkpoints after EVERY graph node, so each node's
progress silently erased the log written during it. A solo run has no second
writer, which is exactly why it looked like it worked.

Now merged with `||`. The keys are disjoint, so the progress object still wins
for everything it owns.

I was wrong about the cause twice before finding this. First I blamed the guest
agent's serial accept loop — real, fixed, and not this. Then I blamed pipe
buffering racing the abort at turn end — plausible, and the drain fix is right on
its own merits, but composed still streamed zero afterwards, which is what ruled
it out. The thing that actually located it was noticing solo and composed differ
by a WRITER, not by a code path.
This commit is contained in:
Omar Sobh
2026-08-07 23:10:27 -07:00
parent 09afa7e7ff
commit 8c93cd8569
+17 -5
View File
@@ -309,20 +309,32 @@ pub async fn claim_next_queued(pool: &PgPool) -> Result<Option<ClaimedTopologyRu
/// Persist mid-run progress: the completed-step checkpoint + journal offset.
/// Touches `updated_at` so the stale-run sweeper treats the job as alive.
///
/// MERGES rather than replaces. This is not cosmetic: a second writer appends
/// live agent output under `checkpoint.log` (see `fleet.rs`, `Uplink::VmOut`),
/// and a composed run checkpoints after EVERY graph node. With `SET checkpoint =
/// $2` each node's progress silently wiped the log written during it, so a
/// composed mission finished with a full `records` array and no output at all —
/// while a solo mission, which has no second writer, streamed fine. The keys are
/// disjoint, so the progress object still wins for everything it owns.
pub async fn checkpoint(
pool: &PgPool,
id: Uuid,
checkpoint: &Value,
last_event_id: i64,
) -> Result<(), DbError> {
sqlx::query!(
// `query` rather than `query!`: the macro verifies against a cached schema
// that would need regenerating for this SQL, and the bind types here are
// unambiguous.
sqlx::query(
"UPDATE topology_runs
SET checkpoint = $2, last_event_id = $3, updated_at = now()
SET checkpoint = COALESCE(checkpoint, '{}'::jsonb) || $2,
last_event_id = $3, updated_at = now()
WHERE id = $1",
id,
checkpoint,
last_event_id,
)
.bind(id)
.bind(checkpoint)
.bind(last_event_id)
.execute(pool)
.await?;
Ok(())