osobh/clawmates
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(skill-use): progressive disclosure, as an arm and not a switch

Browse Source 
Trigger — did the agent reach for the skill when it applied? — cannot be
measured while every body is inlined into the prompt. Nothing was reached
for. `skill_use` has been reporting `NotObservable` for that reason, and it
was right to.

The skills door made retrieval possible; this makes it a delivery arm.
`index` sends each pinned skill's name, description, `when_to_use` and the
uri that returns its body, and the agent fetches what it judges relevant.
`inline` is unchanged and stays the default.

An A/B rather than a switch, because `index` can only cost Compliance: under
`inline` the procedure sits in front of the model whether or not it noticed
it applied. Trading a measured axis for an unmeasured regression in another
is not an improvement, so both arms stay runnable and the arm is recorded on
the mission row.

Three things the mechanism refuses to do:

- `index` without a door falls back to `inline`. An index names bodies and
  says how to fetch them; with no `clawmates_skills` server reachable that is
  a list of dead ends, and it fails as an agent ignoring its skills rather
  than as a missing config. `install_skills_door` now returns whether it
  installed, because the caller needs the answer and not just the log line.

- The scorer reads the arm off the recorded PROMPT, not off the mission row.
  The row says what the mission is configured to do now; the score is being
  computed against a turn that ran then.

- Under `index`, a skill that was offered and never read is a Fail, not the
  inline arm's `NotObservable` — but only where the skill had a checkable
  consequence in that phase. Reusing the inline text would have said "this
  skill was inlined into the prompt" about a skill whose body was never sent,
  and scoring a real miss as a structural blind spot is the failure this
  measurement already made once.

The arm is per mission (`config.skill_delivery`), not only per deployment.
Both arms run against one server process; restarting between them would put a
confound in the comparison that the numbers would not show.

829 tests, 108 binaries, green.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
Omar Sobh
2026-08-25 07:17:55 -05:00
co-authored by Claude Opus 5
parent b58f0347e6
commit f52cff3e04
11 changed files with 735 additions and 48 deletions
Download Patch File Download Diff File Expand all files Collapse all files
crates/cm-api/src/topology_exec.rs
+81 -14
View File
@@ -337,12 +337,50 @@ impl ZeroClawDriveExecutor {
/// for a mission role was unreachable prose, and no measurement of whether
/// skills fire could have returned anything but zero.
///
/// Bodies, not an index. The chat path lists names and lets the claw call
/// `skills.read`; there is no such tool here, so an index would advertise a
/// capability that does not exist — the exact failure this whole change is
/// about. Pinned only (`pin_in_context`), because everything else would go
/// in unbounded and unread.
/// Bodies or an index, depending on the mission's arm — see
/// [`crate::skill_delivery`]. Bodies were once the only honest option:
/// there was no tool on the mission path that could fetch one, so an index
/// would have advertised a capability that did not exist. The skills door
/// changed that, and the arm is now recorded per mission so both can run.
///
/// Pinned only (`pin_in_context`) in either arm, because everything else
/// would go in unbounded and unread.
pub async fn pinned_skills_text(&self, alias: &str) -> Option<String> {
let mode = self.skill_delivery_mode().await;
self.pinned_skills_in_mode(alias, mode).await
}
/// The arm this mission was launched with.
///
/// Read per turn rather than cached on the executor: the executor is
/// constructed from the environment by `topology_worker`, which knows
/// nothing about a mission, and the arm is decided at launch by the code
/// that also learns whether the door installed.
///
/// Anything unreadable — no tap, no row, an unrecognised value — resolves
/// to `Inline`, which is the arm that needs nothing to be true.
pub(crate) async fn skill_delivery_mode(&self) -> crate::skill_delivery::Mode {
let Some(tap) = self.tap.as_ref() else {
return crate::skill_delivery::Mode::Inline;
};
sqlx::query_scalar::<_, Option<String>>(
"SELECT skill_delivery FROM missions WHERE id = $1",
)
.bind(tap.mission_id)
.fetch_optional(&tap.pool)
.await
.ok()
.flatten()
.flatten()
.and_then(|s| crate::skill_delivery::parse(&s))
.unwrap_or(crate::skill_delivery::Mode::Inline)
}
pub(crate) async fn pinned_skills_in_mode(
&self,
alias: &str,
mode: crate::skill_delivery::Mode,
) -> Option<String> {
let tap = self.tap.as_ref()?;
let agent_id = crate::runtime_provision::claw_from_alias(alias)?;
let link = cm_db::repo::agent_template_link::get(&tap.pool, agent_id)
@@ -361,17 +399,29 @@ impl ZeroClawDriveExecutor {
let mut out = String::new();
let mut n = 0usize;
for b in bindings.iter().filter(|b| b.pin_in_context) {
let text = match mode {
crate::skill_delivery::Mode::Inline => b.skill.body.clone(),
// An entry is a few hundred bytes whatever the body weighs, so
// the index arm cannot hit the cap that follows. That is the
// point of it, and the reason the cap is checked against the
// rendered text rather than against the body.
crate::skill_delivery::Mode::Index => crate::skill_delivery::index_entry(
&b.skill.description,
b.skill.when_to_use.as_deref(),
&crate::mcp_skills::skill_uri(b.skill.workspace_id, &b.skill.name),
),
};
// Bounded, and truncation is STATED. A silently clipped procedure
// is worse than an absent one: the agent follows the half it can
// see and reports success against a rule it never read.
if out.len() + b.skill.body.len() > MAX_PINNED_SKILL_BYTES {
if out.len() + text.len() > MAX_PINNED_SKILL_BYTES {
out.push_str(&format!(
"\n[skill \"{}\" omitted — the pinned set exceeded {} bytes]\n",
b.skill.name, MAX_PINNED_SKILL_BYTES
));
continue;
}
out.push_str(&render_pinned_skill(&b.skill.name, &b.skill.body));
out.push_str(&render_pinned_skill(&b.skill.name, &text));
n += 1;
}
if n == 0 {
@@ -778,9 +828,14 @@ impl TurnExecutor for ZeroClawDriveExecutor {
);
fallback
});
// One lookup, used for the section, its preamble and the record.
// Deriving it three times would let a mission compose an index under
// an inline heading if the row changed mid-run.
let mode = self.skill_delivery_mode().await;
let prompt = compose_turn_prompt(
&Self::build_prompt(&req),
self.pinned_skills_text(&alias).await.as_deref(),
self.pinned_skills_in_mode(&alias, mode).await.as_deref(),
mode,
);
// Record what this agent is ACTUALLY about to receive, before driving.
// Re-deriving it later would re-run the skill lookup against a
@@ -795,7 +850,14 @@ impl TurnExecutor for ZeroClawDriveExecutor {
ev.run_id = tap.run_id;
ev.agent_id = crate::runtime_provision::claw_from_alias(&alias);
ev.target = Some(req.role.clone());
ev.detail = serde_json::json!({ "text": prompt, "tier": "container" });
ev.detail = serde_json::json!({
"text": prompt,
"tier": "container",
// The A/B arm, alongside the prompt it produced. `skill_use`
// recovers this from the prompt text itself, so this field is
// for reporting and for catching the two disagreeing.
"skill_delivery": mode.as_str(),
});
crate::mission_events::record(&tap.pool, ev).await;
}
self.drive(&alias, &prompt).await
@@ -807,17 +869,22 @@ impl TurnExecutor for ZeroClawDriveExecutor {
/// Split out from `run_turn` so the wiring is testable: `pinned_skills_text`
/// working and `run_turn` actually calling it are different claims, and the
/// second is the one that was false for every skill in the catalogue.
pub fn compose_turn_prompt(base: &str, skills: Option<&str>) -> String {
pub fn compose_turn_prompt(
base: &str,
skills: Option<&str>,
mode: crate::skill_delivery::Mode,
) -> String {
let Some(skills) = skills.map(str::trim).filter(|s| !s.is_empty()) else {
// No heading when there is nothing under it. An empty "Your skills"
// section tells the model it has skills and then shows it none, which
// is worse than silence.
return base.to_string();
};
format!(
"{base}\n\n# Your skills\n\nThese are procedures you are expected to follow for \
this kind of work. Where one applies to what you are about to do, follow it.\n\n{skills}"
)
// The preamble differs per arm and lives in `skill_delivery`, because it
// is also what the scorer reads the arm back from. Two copies of this
// sentence is two chances for the reader to stop recognising the writer.
let preamble = crate::skill_delivery::preamble(mode);
format!("{base}\n\n# Your skills\n\n{preamble}\n\n{skills}")
}
/// Parse `role=alias,role=alias` into a map (blank/malformed entries skipped).