fix(missions): make an unrunnable test suite legible, and check the runtime at boot
Two changes against the same defect: the platform could not tell a missing capability from a legitimate negative result. verify_tests returned Option<bool>, collapsing four outcomes into None: no suite found, docker unreachable, exec failed, and no exit status. When clawmates-runtime shipped without cargo, every on_green_tests phase returned None and landed on -wip — identical to the reading for "this repo has no tests", which is the conclusion I drew and reported. The gate was correct throughout; it simply could not say why it was unproven. TestOutcome now names the four cases. Gating is unchanged (only Passed clears, unproven is never a pass), and tests_verified keeps its tri-state meaning for existing readers. tests_status and tests_detail are new, so an artifact distinguishes no_suite from could_not_run, and a CouldNotRun is logged as the infrastructure fault it is rather than passing quietly. runtime_preflight probes the runtime container at boot for every tool the platform invokes inside it and names what each absence disables. This is the check that was missing: the Dockerfile gained a toolchain, the image was never built, gw-04 ran the old one for days, and the only symptoms were an ungated suite and a security scan that scanned nothing. A report, not a gate — a missing scanner should stop us believing a scan, not stop the server. Its test guards the probes themselves, since a typo would produce a permanent false "missing" and train operators to ignore it. Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
co-authored by
Claude Opus 5
parent
9bdc3cd89b
commit
f7e336ff5f
@@ -828,3 +828,51 @@ async fn commit_subjects_read_correctly_on_first_pass_and_rerun() {
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/// "No test suite" and "could not run the test suite" must not look alike.
|
||||
///
|
||||
/// verify_tests returned Option<bool>, so both produced `None`. That is how a
|
||||
/// runtime image shipped without `cargo` stayed invisible: every on_green_tests
|
||||
/// phase landed on -wip, which reads exactly like a repository that has no
|
||||
/// tests — the conclusion I drew at the time and reported.
|
||||
///
|
||||
/// Both still gate identically, and that part is deliberate: unproven is not a
|
||||
/// pass, whatever the reason. What changes is that the artifact now says which
|
||||
/// of the two happened, so an infrastructure fault is legible as one.
|
||||
#[tokio::test]
|
||||
async fn an_unrunnable_suite_is_distinguishable_from_no_suite() {
|
||||
use cm_api::mission_delivery::{verify_tests, TestOutcome};
|
||||
|
||||
let tmp = tempfile::tempdir().unwrap();
|
||||
let mission = Uuid::now_v7();
|
||||
let repo = seed_repo(tmp.path(), mission);
|
||||
|
||||
// No Cargo.toml / package.json / pytest markers: nothing to run.
|
||||
let none = verify_tests(&repo, "clawmates-runtime-does-not-exist").await;
|
||||
assert_eq!(none, TestOutcome::NoSuite);
|
||||
assert_eq!(none.status(), "no_suite");
|
||||
assert_eq!(none.verified(), None, "no suite must not clear the gate");
|
||||
|
||||
// A suite exists, but the container named here does not, so it cannot run.
|
||||
std::fs::write(
|
||||
repo.join("Cargo.toml"),
|
||||
"[package]\nname = \"p\"\nversion = \"0.1.0\"\nedition = \"2021\"\n",
|
||||
)
|
||||
.unwrap();
|
||||
let unrunnable = verify_tests(&repo, "clawmates-runtime-does-not-exist").await;
|
||||
assert_eq!(unrunnable.status(), "could_not_run");
|
||||
assert_eq!(
|
||||
unrunnable.verified(),
|
||||
None,
|
||||
"an unrunnable suite must not clear the gate either"
|
||||
);
|
||||
assert!(
|
||||
unrunnable.detail().is_some_and(|d| !d.is_empty()),
|
||||
"an infrastructure fault must carry its reason into the artifact"
|
||||
);
|
||||
assert_ne!(
|
||||
unrunnable.status(),
|
||||
none.status(),
|
||||
"the two must be distinguishable — this is the whole point"
|
||||
);
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user