fix(missions): stop three launch failures from passing as success

A verification run against the deployed stack found a chain mission whose
phase 0 reported `completed` with zero files, no commit error and no push
error — indistinguishable from a phase that correctly had nothing to do.
Three separate defects had to line up, each of them the same shape: a
failure sharing its representation with a legitimate negative result.

1. `pin_agent_workspaces` embedded the whole config in one `sh -c` argv.
   That works until the file grows — config gains a block per provisioned
   claw — then fails with `argument list too long`. Now written through
   the tar upload API, which has no argv limit, so the failure mode is
   gone rather than merely further away.

2. A failed pin was logged "(continuing)". Without the pin, agents write
   to their sandboxes and the committer finds nothing in /mission/repo —
   the mission cannot deliver, so the launch now fails where someone is
   still looking. The restart that applies the pin is fatal for the same
   reason.

3. `capture_phase_diff_at` swallowed `git diff` failures with
   `unwrap_or_default`, so an unreadable base landed `empty: true,
   files_changed: 0` — byte-identical to an honest no-op. The error is now
   recorded as `diff_error`, and an empty patch that came from a failed
   diff is no longer trusted to mean an unchanged tree.

Adds scripts/verify-mission-delivery.sh, which found #1 and #2 on its
first real run. Its probes are fail-closed: no placeholder values, a
self-test that proves the uid probe can detect the split it looks for,
and FAIL-NORUN for a scenario that never executed. Its own first version
had this bug too — a `die` inside `$(...)` exited the subshell, so a run
that could not authenticate printed "all checks passed" and exited 0.

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
Omar Sobh
2026-08-04 18:01:02 -07:00
co-authored by Claude Opus 5
parent bb274d08c6
commit 1253595ba7
5 changed files with 597 additions and 38 deletions
+87
View File
@@ -90,6 +90,60 @@ pub async fn copy_in(
.map_err(|e| format!("copy into {container}:{CONTAINER_MISSION_DIR}: {e}"))
}
/// Build a one-entry tar. Split out from [`put_file`] so the size-independence
/// that is the whole point can be tested without Docker.
fn single_file_archive(name: &str, contents: &[u8]) -> Result<Vec<u8>, String> {
let mut header = tar::Header::new_gnu();
header
.set_path(name)
.map_err(|e| format!("tar path {name}: {e}"))?;
header.set_size(contents.len() as u64);
header.set_mode(0o600);
header.set_entry_type(tar::EntryType::Regular);
header.set_cksum();
let mut builder = tar::Builder::new(Vec::new());
builder
.append(&header, contents)
.map_err(|e| format!("tar {name}: {e}"))?;
builder
.into_inner()
.map_err(|e| format!("finish archive for {name}: {e}"))
}
/// Write one file into a container, at any size.
///
/// The obvious way to do this is `sh -c "printf … > file"`, and it works right
/// up until the payload approaches `ARG_MAX`, at which point exec fails with
/// `argument list too long`. That is a size-dependent failure in a code path
/// whose payload grows with use, which makes it a bug that ships green and
/// surfaces in production — as it did, silently unpinning every agent in
/// mission `019fcf62`. Tar has no argv limit.
///
/// The write is not atomic. Callers that need it can upload beside the target
/// and rename; the config writer does not, because the daemon reads its config
/// once at boot and is restarted afterwards.
pub async fn put_file(
docker: &Docker,
container: &str,
path: &str,
contents: &[u8],
) -> Result<(), String> {
let (dir, file) = path
.rsplit_once('/')
.ok_or_else(|| format!("{path} is not an absolute path"))?;
let dir = if dir.is_empty() { "/" } else { dir };
let archive = single_file_archive(file, contents)?;
let opts = bollard::query_parameters::UploadToContainerOptionsBuilder::default()
.path(dir)
.build();
docker
.upload_to_container(container, Some(opts), bollard::body_full(archive.into()))
.await
.map_err(|e| format!("upload {path} to {container}: {e}"))
}
/// Copy a directory back out of a container onto the host.
pub async fn copy_out(
docker: &Docker,
@@ -260,6 +314,39 @@ mod tests {
}
}
/// The regression this exists for: a config large enough to blow `ARG_MAX`
/// via `sh -c` must round-trip untouched. 2 MB is well past the ~128 KB
/// limit that unpinned every agent in mission `019fcf62`.
#[test]
fn a_file_far_past_arg_max_round_trips() {
let big = "workspace_path = \"/mission/repo\"\n".repeat(64 * 1024);
assert!(big.len() > 2_000_000, "the fixture must exceed ARG_MAX");
let archive = single_file_archive("config.toml", big.as_bytes()).unwrap();
let tmp = tempfile::tempdir().unwrap();
unpack_into(&archive, tmp.path()).unwrap();
assert_eq!(
std::fs::read_to_string(tmp.path().join("config.toml")).unwrap(),
big,
"a large config must survive byte-for-byte"
);
}
/// TOML holding quotes, newlines and backslashes went through a shell
/// before; nothing may depend on quoting now.
#[test]
fn shell_metacharacters_survive_the_archive() {
let nasty = "path = \"/a'b\\\"c\"\n$(rm -rf /) `id` \\\\ \n";
let archive = single_file_archive("config.toml", nasty.as_bytes()).unwrap();
let tmp = tempfile::tempdir().unwrap();
unpack_into(&archive, tmp.path()).unwrap();
assert_eq!(
std::fs::read_to_string(tmp.path().join("config.toml")).unwrap(),
nasty
);
}
#[test]
fn an_empty_directory_packs_without_error() {
let tmp = tempfile::tempdir().unwrap();