feat(skill-use): progressive disclosure, as an arm and not a switch

Trigger — did the agent reach for the skill when it applied? — cannot be
measured while every body is inlined into the prompt. Nothing was reached
for. `skill_use` has been reporting `NotObservable` for that reason, and it
was right to.

The skills door made retrieval possible; this makes it a delivery arm.
`index` sends each pinned skill's name, description, `when_to_use` and the
uri that returns its body, and the agent fetches what it judges relevant.
`inline` is unchanged and stays the default.

An A/B rather than a switch, because `index` can only cost Compliance: under
`inline` the procedure sits in front of the model whether or not it noticed
it applied. Trading a measured axis for an unmeasured regression in another
is not an improvement, so both arms stay runnable and the arm is recorded on
the mission row.

Three things the mechanism refuses to do:

- `index` without a door falls back to `inline`. An index names bodies and
  says how to fetch them; with no `clawmates_skills` server reachable that is
  a list of dead ends, and it fails as an agent ignoring its skills rather
  than as a missing config. `install_skills_door` now returns whether it
  installed, because the caller needs the answer and not just the log line.

- The scorer reads the arm off the recorded PROMPT, not off the mission row.
  The row says what the mission is configured to do now; the score is being
  computed against a turn that ran then.

- Under `index`, a skill that was offered and never read is a Fail, not the
  inline arm's `NotObservable` — but only where the skill had a checkable
  consequence in that phase. Reusing the inline text would have said "this
  skill was inlined into the prompt" about a skill whose body was never sent,
  and scoring a real miss as a structural blind spot is the failure this
  measurement already made once.

The arm is per mission (`config.skill_delivery`), not only per deployment.
Both arms run against one server process; restarting between them would put a
confound in the comparison that the numbers would not show.

829 tests, 108 binaries, green.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
Omar Sobh
2026-08-25 07:17:55 -05:00
co-authored by Claude Opus 5
parent b58f0347e6
commit f52cff3e04
11 changed files with 735 additions and 48 deletions
+50 -9
View File
@@ -349,7 +349,7 @@ pub async fn on_launch(
// Only when this mission got its OWN container — the shared runtime is
// not ours to reconfigure, and `mission_gateway` being Some is exactly
// the signal that `ensure_container` ran.
if mission_gateway.is_some() {
let door = if mission_gateway.is_some() {
install_skills_door(
pool,
user_id,
@@ -357,8 +357,22 @@ pub async fn on_launch(
&crate::mission_runtime::container_name(mission_id),
p,
)
.await;
}
.await
} else {
false
};
// Decided here and recorded, not re-derived per turn: this is the only
// point that knows whether the door actually installed, and an arm that
// could change mid-mission would make the run unattributable.
record_skill_delivery(
pool,
mission_id,
crate::skill_delivery::resolve(
crate::skill_delivery::requested_for(&mission.config),
door,
),
)
.await;
}
let mut first_team_id: Option<Uuid> = None;
let mut provisioned_claws: Vec<cm_domain::AgentId> = Vec::new();
@@ -916,19 +930,23 @@ fn default_accent_for(slot: &str) -> &'static str {
///
/// Every failure degrades to "no door", never to a failed launch. A mission
/// that cannot retrieve a skill still delivers.
/// Returns whether the door is installed AND reachable. The caller needs the
/// answer, not just the log line: the `index` delivery arm hands agents a list
/// of uris to fetch, and without a door every one of them is a dead end that
/// reads as an agent ignoring its skills.
async fn install_skills_door(
pool: &PgPool,
user_id: cm_domain::UserId,
mission_id: Uuid,
container: &str,
prov: &RuntimeProvisioner,
) {
) -> bool {
let Some(origin) = crate::container_tool_hooks::api_origin() else {
eprintln!(
"mission_orchestrator: no API origin for the skills door (set \
CLAWMATES_API_ORIGIN) — mission {mission_id} runs without it"
);
return;
return false;
};
// Outlives the longest mission we have seen, and expires on its own so a
// leaked container does not leave a live credential behind indefinitely.
@@ -943,31 +961,54 @@ async fn install_skills_door(
"mission_orchestrator: could not mint a skills token ({e}) — \
mission {mission_id} runs without the door"
);
return;
return false;
}
};
let docker = match crate::container_exec::connect() {
Ok(d) => d,
Err(e) => {
eprintln!("mission_orchestrator: cannot reach docker for the skills door: {e}");
return;
return false;
}
};
let doc = crate::container_tool_hooks::mcp_document(&origin, &token);
let Some(path) = crate::container_tool_hooks::install_door(&docker, container, &doc).await
else {
// `install_door` already said why.
return;
return false;
};
if let Err(e) = prov.set_claude_cli_mcp_config(&path).await {
eprintln!(
"mission_orchestrator: wrote the MCP config but could not point \
claude_cli at it ({e}) — the door is installed and unreachable"
);
return;
return false;
}
eprintln!(
"mission_orchestrator: skills door installed for mission {mission_id} \
({origin}/mcp/skills)"
);
true
}
/// Record which arm this mission runs, so every turn composes the same one and
/// the score can be attributed to it afterwards.
///
/// A write failure is not fatal: `skill_delivery_mode` reads NULL as `inline`,
/// which is the arm that needs nothing installed. A mission that quietly ran
/// the control arm is a lost data point; a mission that failed to launch over
/// a telemetry column is a lost mission.
async fn record_skill_delivery(pool: &PgPool, mission_id: Uuid, mode: crate::skill_delivery::Mode) {
if let Err(e) = sqlx::query("UPDATE missions SET skill_delivery = $2 WHERE id = $1")
.bind(mission_id)
.bind(mode.as_str())
.execute(pool)
.await
{
eprintln!(
"mission_orchestrator: could not record skill_delivery={} for mission \
{mission_id} ({e}) — its turns will compose skills inline",
mode.as_str()
);
}
}