feat(skills): a skill that must be read cannot be left to be noticed
The `index` arm hands an agent a list of uris and trusts it to fetch what applies. Measured on the first A/B pair, that is mostly what happens — each agent fetched the skill bound to its own role and no other, which is the result that made Trigger observable at all. `workspace-repo-commit-protocol` is the case it fails on. It scored Trigger=FAIL beside a PASSING boundary check: the rule was live and unread. A procedure that applies to everyone who writes reads as nobody's in particular, so no agent recognises it as theirs and no agent fetches it. Upstream ZeroClaw arrived at the same place from the other direction and gave its compact injection mode an `always: true` frontmatter escape hatch (#9520). This is that hatch as a column: `skills.always_inject`, default FALSE, so nothing changes for an existing skill and the inline arm is untouched either way. Two halves, because delivering it and scoring it are different mistakes: - Delivery: under `Index`, an `always_inject` skill renders its BODY. - Scoring: the arm belongs to the PROMPT and `always_inject` belongs to the SKILL, so the scorer now asks per skill which one it got. A skill whose body is in the prompt was handed over, and a Trigger miss cannot be charged against an agent that was never asked to fetch anything. `skill_was_indexed` reads that off the rendered prompt via `READ_IT`, a constant now shared with `index_entry` — two spellings of one marker is how a detector quietly stops detecting. Suite: 108 binaries, 840 tests, green. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01WZb5A2kfVfjpdwSochkuHz
This commit is contained in:
co-authored by
Claude Opus 5
parent
563b074116
commit
2f1a870949
@@ -263,10 +263,19 @@ pub fn score(
|
||||
.into_iter()
|
||||
.map(|skill| {
|
||||
let (compliance, boundary) = check(&skill, evidence);
|
||||
// The arm belongs to the prompt; `always_inject` belongs to the
|
||||
// skill. A skill whose BODY is in the prompt was handed over, so
|
||||
// there is no reaching-for to observe even under `Index` — scoring
|
||||
// it as a Trigger miss would report a failure against an agent that
|
||||
// was never asked to fetch anything.
|
||||
let delivered = match crate::skill_delivery::skill_was_indexed(prompt, &skill) {
|
||||
Some(false) => crate::skill_delivery::Mode::Inline,
|
||||
_ => mode,
|
||||
};
|
||||
SkillUse {
|
||||
source_kind: source_kinds(&skill),
|
||||
trigger: trigger_verdict(
|
||||
mode,
|
||||
delivered,
|
||||
retrieved.contains(&skill),
|
||||
&compliance,
|
||||
&boundary,
|
||||
@@ -1007,6 +1016,62 @@ mod tests {
|
||||
}
|
||||
|
||||
/// The control arm has to be unchanged, or the A/B measures this edit too.
|
||||
/// The case this whole flag exists for.
|
||||
///
|
||||
/// `workspace-repo-commit-protocol` applies to every agent that writes,
|
||||
/// which is exactly why no agent reads it as *theirs* — it scored
|
||||
/// Trigger=FAIL beside a passing boundary check on the first A/B pair.
|
||||
/// Marked `always_inject`, its body is in the prompt under the index arm,
|
||||
/// and a skill the agent was handed cannot be a reaching-for failure.
|
||||
#[test]
|
||||
fn a_skill_delivered_in_full_under_the_index_arm_is_not_a_trigger_miss() {
|
||||
let prompt = format!(
|
||||
"{}\n{}{}",
|
||||
crate::skill_delivery::INDEX_PREAMBLE,
|
||||
crate::topology_exec::render_pinned_skill(
|
||||
"workspace-repo-commit-protocol",
|
||||
"Commit only inside /workspace/repo. Never write outside it.",
|
||||
),
|
||||
rendered_index(&[("arxiv-daily", "when sweeping arxiv")]),
|
||||
);
|
||||
let tools = [];
|
||||
let ev = Evidence { text: "", tools: &tools };
|
||||
let got = score(&prompt, &ev, &|_| "builtin".to_string());
|
||||
let it = got
|
||||
.iter()
|
||||
.find(|u| u.skill == "workspace-repo-commit-protocol")
|
||||
.expect("the always-injected skill must still be scored");
|
||||
assert!(
|
||||
matches!(it.trigger, Verdict::NotObservable(_)),
|
||||
"handed over, not offered — there is no retrieval to miss: {:?}",
|
||||
it.trigger
|
||||
);
|
||||
}
|
||||
|
||||
/// The flag must not leak: a skill still delivered as an index entry keeps
|
||||
/// being scored on whether it was fetched.
|
||||
#[test]
|
||||
fn an_indexed_skill_in_the_same_prompt_is_still_judged_on_retrieval() {
|
||||
let prompt = format!(
|
||||
"{}\n{}{}",
|
||||
crate::skill_delivery::INDEX_PREAMBLE,
|
||||
crate::topology_exec::render_pinned_skill(
|
||||
"workspace-repo-commit-protocol",
|
||||
"Commit only inside /workspace/repo.",
|
||||
),
|
||||
rendered_index(&[("arxiv-daily", "when sweeping arxiv")]),
|
||||
);
|
||||
for u in score(&prompt, &Evidence { text: "", tools: &[] }, &|_| "builtin".into()) {
|
||||
if u.skill != "workspace-repo-commit-protocol" {
|
||||
assert!(
|
||||
!matches!(u.trigger, Verdict::NotObservable(_)),
|
||||
"{} was offered by uri, so retrieval is observable for it",
|
||||
u.skill
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_inline_arm_still_reports_trigger_as_unobservable() {
|
||||
let prompt = rendered(&[("workspace-repo-commit-protocol", "body")]);
|
||||
|
||||
Reference in New Issue
Block a user