feat(skill-use): progressive disclosure, as an arm and not a switch
Trigger — did the agent reach for the skill when it applied? — cannot be measured while every body is inlined into the prompt. Nothing was reached for. `skill_use` has been reporting `NotObservable` for that reason, and it was right to. The skills door made retrieval possible; this makes it a delivery arm. `index` sends each pinned skill's name, description, `when_to_use` and the uri that returns its body, and the agent fetches what it judges relevant. `inline` is unchanged and stays the default. An A/B rather than a switch, because `index` can only cost Compliance: under `inline` the procedure sits in front of the model whether or not it noticed it applied. Trading a measured axis for an unmeasured regression in another is not an improvement, so both arms stay runnable and the arm is recorded on the mission row. Three things the mechanism refuses to do: - `index` without a door falls back to `inline`. An index names bodies and says how to fetch them; with no `clawmates_skills` server reachable that is a list of dead ends, and it fails as an agent ignoring its skills rather than as a missing config. `install_skills_door` now returns whether it installed, because the caller needs the answer and not just the log line. - The scorer reads the arm off the recorded PROMPT, not off the mission row. The row says what the mission is configured to do now; the score is being computed against a turn that ran then. - Under `index`, a skill that was offered and never read is a Fail, not the inline arm's `NotObservable` — but only where the skill had a checkable consequence in that phase. Reusing the inline text would have said "this skill was inlined into the prompt" about a skill whose body was never sent, and scoring a real miss as a structural blind spot is the failure this measurement already made once. The arm is per mission (`config.skill_delivery`), not only per deployment. Both arms run against one server process; restarting between them would put a confound in the comparison that the numbers would not show. 829 tests, 108 binaries, green. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
Omar Sobh
co-authored by
Claude Opus 5
Claude Opus 5
parent
b58f0347e6
commit
f52cff3e04
@@ -10,6 +10,7 @@
|
||||
//! structurally unreadable. That is why this is a test and not a comment: the
|
||||
//! failure produced no error anywhere, and every layer reported success.
|
||||
|
||||
use cm_api::skill_delivery::Mode;
|
||||
use cm_api::topology_exec::{MissionTap, ZeroClawDriveExecutor};
|
||||
use cm_domain::{Agent, AgentId, AgentStatus, Role, User, UserId, Workspace, WorkspaceId};
|
||||
|
||||
@@ -121,6 +122,14 @@ async fn seed_claw_with_pinned_skill(pool: &sqlx::PgPool, body: &str) -> (String
|
||||
}
|
||||
|
||||
fn executor(pool: &sqlx::PgPool, workspace_id: WorkspaceId) -> ZeroClawDriveExecutor {
|
||||
executor_for(pool, workspace_id, Uuid::now_v7())
|
||||
}
|
||||
|
||||
fn executor_for(
|
||||
pool: &sqlx::PgPool,
|
||||
workspace_id: WorkspaceId,
|
||||
mission_id: Uuid,
|
||||
) -> ZeroClawDriveExecutor {
|
||||
ZeroClawDriveExecutor::new(
|
||||
"http://127.0.0.1:1".into(),
|
||||
"unused".into(),
|
||||
@@ -130,12 +139,93 @@ fn executor(pool: &sqlx::PgPool, workspace_id: WorkspaceId) -> ZeroClawDriveExec
|
||||
.with_tap(MissionTap {
|
||||
pool: pool.clone(),
|
||||
workspace_id: workspace_id.as_uuid(),
|
||||
mission_id: Uuid::now_v7(),
|
||||
mission_id,
|
||||
phase_id: None,
|
||||
run_id: None,
|
||||
})
|
||||
}
|
||||
|
||||
/// A mission row carrying an explicit delivery arm.
|
||||
async fn seed_mission_on_arm(pool: &sqlx::PgPool, ws: WorkspaceId, arm: Option<&str>) -> Uuid {
|
||||
let mission = Uuid::now_v7();
|
||||
sqlx::query(
|
||||
"INSERT INTO missions (id, workspace_id, title, template_kind, status, skill_delivery)
|
||||
VALUES ($1, $2, 'arm', 'research_only', 'running', $3)",
|
||||
)
|
||||
.bind(mission)
|
||||
.bind(ws.as_uuid())
|
||||
.bind(arm)
|
||||
.execute(pool)
|
||||
.await
|
||||
.unwrap();
|
||||
mission
|
||||
}
|
||||
|
||||
// ── The index arm ───────────────────────────────────────────────────
|
||||
//
|
||||
// Trigger — did the agent reach for the skill? — cannot exist while every body
|
||||
// is handed over unasked. These tests cover the arm that makes it a question,
|
||||
// and the one guarantee the control arm needs: that it did not change.
|
||||
|
||||
#[tokio::test]
|
||||
async fn the_index_arm_sends_the_uri_and_withholds_the_body() {
|
||||
let pool = cm_testkit::test_pool().await;
|
||||
const MARKER: &str = "Never review a paper from its title alone.";
|
||||
let (alias, ws) = seed_claw_with_pinned_skill(&pool, MARKER).await;
|
||||
let mission = seed_mission_on_arm(&pool, ws, Some("index")).await;
|
||||
|
||||
let text = executor_for(&pool, ws, mission)
|
||||
.pinned_skills_text(&alias)
|
||||
.await
|
||||
.expect("an indexed skill is still delivered — as an entry, not a body");
|
||||
|
||||
assert!(
|
||||
!text.contains(MARKER),
|
||||
"the BODY is what the index withholds; leaving it in delivers both \
|
||||
arms at once and measures neither. Got:\n{text}"
|
||||
);
|
||||
assert!(
|
||||
text.contains("uri=\"skill:global/delivery-test-skill-"),
|
||||
"an entry without a fetchable uri names a procedure the agent cannot \
|
||||
obtain — worse than inlining it. Got:\n{text}"
|
||||
);
|
||||
assert!(
|
||||
text.contains("When to use: always"),
|
||||
"`when_to_use` is the only thing the agent can judge relevance from, \
|
||||
and judging relevance is the entire axis. Got:\n{text}"
|
||||
);
|
||||
// The scorer counts delivered skills by the marker, in both arms.
|
||||
assert_eq!(
|
||||
cm_api::topology_exec::skill_names_in(&text).len(),
|
||||
1,
|
||||
"an indexed skill must still count as delivered:\n{text}"
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn an_unrecorded_arm_delivers_bodies() {
|
||||
let pool = cm_testkit::test_pool().await;
|
||||
const MARKER: &str = "Never review a paper from its title alone.";
|
||||
let (alias, ws) = seed_claw_with_pinned_skill(&pool, MARKER).await;
|
||||
|
||||
// Every mission that ran before the column existed, plus any row whose
|
||||
// value is unreadable, plus a turn with no mission row at all. All three
|
||||
// resolve to the arm that needs nothing installed to work.
|
||||
for arm in [None, Some("nonsense")] {
|
||||
let mission = seed_mission_on_arm(&pool, ws, arm).await;
|
||||
let text = executor_for(&pool, ws, mission)
|
||||
.pinned_skills_text(&alias)
|
||||
.await
|
||||
.unwrap();
|
||||
assert!(
|
||||
text.contains(MARKER),
|
||||
"skill_delivery={arm:?} must deliver the body — an index arm \
|
||||
selected by accident hands out uris behind a door that may not \
|
||||
be installed. Got:\n{text}"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn a_pinned_skill_body_reaches_the_turn_prompt() {
|
||||
let pool = cm_testkit::test_pool().await;
|
||||
@@ -149,9 +239,9 @@ async fn a_pinned_skill_body_reaches_the_turn_prompt() {
|
||||
|
||||
assert!(
|
||||
text.contains(MARKER),
|
||||
"the skill BODY must be present, not just its name — there is no \
|
||||
`skills.read` tool on the mission path, so an index would name a \
|
||||
procedure the agent has no way to fetch. Got:\n{text}"
|
||||
"the default arm delivers the BODY. It is the control in the delivery \
|
||||
A/B, so it has to stay what production has always sent; the index arm \
|
||||
is selected per mission and tested separately. Got:\n{text}"
|
||||
);
|
||||
}
|
||||
|
||||
@@ -197,13 +287,13 @@ async fn an_agent_with_no_pinned_skills_adds_nothing() {
|
||||
fn the_composed_prompt_carries_the_skill_and_omits_the_heading_when_empty() {
|
||||
let base = "You are the \"researcher\" agent.\n\nTask: read the papers";
|
||||
|
||||
let with = cm_api::topology_exec::compose_turn_prompt(base, Some("## arxiv-daily\nDo not re-search."));
|
||||
let with = cm_api::topology_exec::compose_turn_prompt(base, Some("## arxiv-daily\nDo not re-search."), Mode::Inline);
|
||||
assert!(with.contains("Task: read the papers"), "the base turn must survive");
|
||||
assert!(with.contains("Do not re-search."), "the skill body must be in the prompt");
|
||||
assert!(with.contains("# Your skills"), "the section needs a heading");
|
||||
|
||||
for empty in [None, Some(""), Some(" \n ")] {
|
||||
let without = cm_api::topology_exec::compose_turn_prompt(base, empty);
|
||||
let without = cm_api::topology_exec::compose_turn_prompt(base, empty, Mode::Inline);
|
||||
assert_eq!(
|
||||
without, base,
|
||||
"with no skills the prompt must be byte-identical to the base — an \
|
||||
@@ -277,7 +367,7 @@ async fn the_microvm_and_session_tiers_get_the_skill_in_their_task_text() {
|
||||
);
|
||||
|
||||
// All three tiers share this composition, so testing it once covers them.
|
||||
let composed = cm_api::topology_exec::compose_turn_prompt("Task: read the papers", Some(&skills));
|
||||
let composed = cm_api::topology_exec::compose_turn_prompt("Task: read the papers", Some(&skills), Mode::Inline);
|
||||
assert!(composed.contains(MARKER));
|
||||
assert!(composed.contains("Task: read the papers"));
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user