fix(missions): make an unrunnable test suite legible, and check the runtime at boot
ci / gates (push) Failing after 6s
ci / rust (push) Skipped
ci / frontend (push) Skipped
ci / e2e (push) Skipped
ci / publish (push) Skipped

Two changes against the same defect: the platform could not tell a missing
capability from a legitimate negative result.

verify_tests returned Option<bool>, collapsing four outcomes into None:
no suite found, docker unreachable, exec failed, and no exit status. When
clawmates-runtime shipped without cargo, every on_green_tests phase
returned None and landed on -wip — identical to the reading for "this
repo has no tests", which is the conclusion I drew and reported. The gate
was correct throughout; it simply could not say why it was unproven.

TestOutcome now names the four cases. Gating is unchanged (only Passed
clears, unproven is never a pass), and tests_verified keeps its tri-state
meaning for existing readers. tests_status and tests_detail are new, so an
artifact distinguishes no_suite from could_not_run, and a CouldNotRun is
logged as the infrastructure fault it is rather than passing quietly.

runtime_preflight probes the runtime container at boot for every tool the
platform invokes inside it and names what each absence disables. This is
the check that was missing: the Dockerfile gained a toolchain, the image
was never built, gw-04 ran the old one for days, and the only symptoms
were an ungated suite and a security scan that scanned nothing. A report,
not a gate — a missing scanner should stop us believing a scan, not stop
the server. Its test guards the probes themselves, since a typo would
produce a permanent false "missing" and train operators to ignore it.

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
Omar Sobh
2026-08-02 17:22:47 -07:00
co-authored by Claude Opus 5
parent 9bdc3cd89b
commit f7e336ff5f
5 changed files with 350 additions and 16 deletions
+48
View File
@@ -828,3 +828,51 @@ async fn commit_subjects_read_correctly_on_first_pass_and_rerun() {
);
}
}
/// "No test suite" and "could not run the test suite" must not look alike.
///
/// verify_tests returned Option<bool>, so both produced `None`. That is how a
/// runtime image shipped without `cargo` stayed invisible: every on_green_tests
/// phase landed on -wip, which reads exactly like a repository that has no
/// tests — the conclusion I drew at the time and reported.
///
/// Both still gate identically, and that part is deliberate: unproven is not a
/// pass, whatever the reason. What changes is that the artifact now says which
/// of the two happened, so an infrastructure fault is legible as one.
#[tokio::test]
async fn an_unrunnable_suite_is_distinguishable_from_no_suite() {
use cm_api::mission_delivery::{verify_tests, TestOutcome};
let tmp = tempfile::tempdir().unwrap();
let mission = Uuid::now_v7();
let repo = seed_repo(tmp.path(), mission);
// No Cargo.toml / package.json / pytest markers: nothing to run.
let none = verify_tests(&repo, "clawmates-runtime-does-not-exist").await;
assert_eq!(none, TestOutcome::NoSuite);
assert_eq!(none.status(), "no_suite");
assert_eq!(none.verified(), None, "no suite must not clear the gate");
// A suite exists, but the container named here does not, so it cannot run.
std::fs::write(
repo.join("Cargo.toml"),
"[package]\nname = \"p\"\nversion = \"0.1.0\"\nedition = \"2021\"\n",
)
.unwrap();
let unrunnable = verify_tests(&repo, "clawmates-runtime-does-not-exist").await;
assert_eq!(unrunnable.status(), "could_not_run");
assert_eq!(
unrunnable.verified(),
None,
"an unrunnable suite must not clear the gate either"
);
assert!(
unrunnable.detail().is_some_and(|d| !d.is_empty()),
"an infrastructure fault must carry its reason into the artifact"
);
assert_ne!(
unrunnable.status(),
none.status(),
"the two must be distinguishable — this is the whole point"
);
}