fix(runs): the composed worker's checkpoint wiped the live log on every node
Composed missions streamed ZERO bytes while solo missions streamed fine. Same executor, same command, same guest — `HubVms::run` is a straight passthrough — and the node logged a tail starting for all five graph nodes against the correct outer run id, with no errors. The bytes simply were not there at the end. Two writers, one column. `fleet.rs` appends live output under `checkpoint.log`; `topology_runs::checkpoint` wrote `SET checkpoint = $2`, replacing the whole object. A composed run checkpoints after EVERY graph node, so each node's progress silently erased the log written during it. A solo run has no second writer, which is exactly why it looked like it worked. Now merged with `||`. The keys are disjoint, so the progress object still wins for everything it owns. I was wrong about the cause twice before finding this. First I blamed the guest agent's serial accept loop — real, fixed, and not this. Then I blamed pipe buffering racing the abort at turn end — plausible, and the drain fix is right on its own merits, but composed still streamed zero afterwards, which is what ruled it out. The thing that actually located it was noticing solo and composed differ by a WRITER, not by a code path.
This commit is contained in:
@@ -309,20 +309,32 @@ pub async fn claim_next_queued(pool: &PgPool) -> Result<Option<ClaimedTopologyRu
|
||||
|
||||
/// Persist mid-run progress: the completed-step checkpoint + journal offset.
|
||||
/// Touches `updated_at` so the stale-run sweeper treats the job as alive.
|
||||
///
|
||||
/// MERGES rather than replaces. This is not cosmetic: a second writer appends
|
||||
/// live agent output under `checkpoint.log` (see `fleet.rs`, `Uplink::VmOut`),
|
||||
/// and a composed run checkpoints after EVERY graph node. With `SET checkpoint =
|
||||
/// $2` each node's progress silently wiped the log written during it, so a
|
||||
/// composed mission finished with a full `records` array and no output at all —
|
||||
/// while a solo mission, which has no second writer, streamed fine. The keys are
|
||||
/// disjoint, so the progress object still wins for everything it owns.
|
||||
pub async fn checkpoint(
|
||||
pool: &PgPool,
|
||||
id: Uuid,
|
||||
checkpoint: &Value,
|
||||
last_event_id: i64,
|
||||
) -> Result<(), DbError> {
|
||||
sqlx::query!(
|
||||
// `query` rather than `query!`: the macro verifies against a cached schema
|
||||
// that would need regenerating for this SQL, and the bind types here are
|
||||
// unambiguous.
|
||||
sqlx::query(
|
||||
"UPDATE topology_runs
|
||||
SET checkpoint = $2, last_event_id = $3, updated_at = now()
|
||||
SET checkpoint = COALESCE(checkpoint, '{}'::jsonb) || $2,
|
||||
last_event_id = $3, updated_at = now()
|
||||
WHERE id = $1",
|
||||
id,
|
||||
checkpoint,
|
||||
last_event_id,
|
||||
)
|
||||
.bind(id)
|
||||
.bind(checkpoint)
|
||||
.bind(last_event_id)
|
||||
.execute(pool)
|
||||
.await?;
|
||||
Ok(())
|
||||
|
||||
Reference in New Issue
Block a user