feat(skills): a files arm — progressive disclosure through Read, not a deferred tool
The `index` arm retrieves through `ReadMcpResourceTool`, which is DEFERRED:
absent from the agent's default tool list until `ToolSearch` loads it. Across
three matched production runs (same recipe, same task, same three offered
uris) it retrieved 1 skill in 9 chances:
01a07812 delegation forced no instruction 0/3
01a0842e no delegation no instruction 1/3
01a09877 no delegation told to load it 0/3
The third run is the decisive one. The preamble said in plain words to run
ToolSearch first; all three prompts carried it; zero ToolSearch calls, and the
three reasoning narratives never mention skills at all. The section was not
declined, it was never engaged with. Instruction is not the lever.
`Read` is a core tool. Never deferred, and every one of those agents used it.
So this arm keeps progressive disclosure exactly as `index` has it — a name, a
`when_to_use`, and a pointer the agent has to follow — and changes only what
the pointer is: a path under /mission/skills instead of an MCP uri. The bodies
are written into the container at launch (every visible skill, one tar upload;
bindings resolve per agent at turn time so a per-mission subset is not knowable
here) and a `Read` of that path is a tapped tool call, so Trigger is exactly as
observable as before.
A third arm and not a replacement, selected per mission like the others, so
the comparison runs against one binary. `resolve` falls back to `inline` when
the files were not written, for the reason `index` does: a pointer to nothing
reads as an agent ignoring its skills.
The writer and reader of a path are one pair of functions
(`skill_file_path` / `skill_from_file_path`), matched by the scorer through
the same seam `parse_uri` uses, and the end-to-end test fails when the matcher
is broken. `Mode::is_retrieval` exists so the next arm cannot silently inherit
`inline`'s "not observable" for what is a miss.
Co-Authored-By: Claude Opus 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01WZb5A2kfVfjpdwSochkuHz
This commit is contained in:
co-authored by
Claude Opus 5
parent
8d6310f126
commit
00160739de
+111
-12
@@ -174,13 +174,26 @@ impl<'a> Evidence<'a> {
|
||||
pub fn retrieved_skills(ev: &Evidence<'_>) -> Vec<String> {
|
||||
let mut out = Vec::new();
|
||||
for t in ev.tools {
|
||||
if t.tool != "ReadMcpResourceTool" {
|
||||
continue;
|
||||
}
|
||||
let Some(uri) = t.input.get("uri").and_then(|v| v.as_str()) else {
|
||||
continue;
|
||||
let name = match t.tool.as_str() {
|
||||
"ReadMcpResourceTool" => t
|
||||
.input
|
||||
.get("uri")
|
||||
.and_then(|v| v.as_str())
|
||||
.and_then(crate::mcp_skills::parse_uri)
|
||||
.map(|(_, name)| name),
|
||||
// The `files` arm: the pointer is a path and the retrieval is a
|
||||
// plain `Read`. Matched through `skill_from_file_path`, the reader
|
||||
// half of the function that wrote the path, for the same reason
|
||||
// the uri goes through `parse_uri`. A `Read` anywhere else is an
|
||||
// ordinary file read and is not a retrieval of anything.
|
||||
"Read" => t
|
||||
.input
|
||||
.get("file_path")
|
||||
.and_then(|v| v.as_str())
|
||||
.and_then(crate::skill_delivery::skill_from_file_path),
|
||||
_ => None,
|
||||
};
|
||||
if let Some((_, name)) = crate::mcp_skills::parse_uri(uri) {
|
||||
if let Some(name) = name {
|
||||
if !out.contains(&name) {
|
||||
out.push(name);
|
||||
}
|
||||
@@ -216,12 +229,12 @@ fn trigger_verdict(
|
||||
through the door instead and this becomes a tool call"
|
||||
.into(),
|
||||
),
|
||||
crate::skill_delivery::Mode::Index => {
|
||||
// Offered by name and `when_to_use`, and never opened. Whether that
|
||||
// is a miss depends on whether the skill had anything to say about
|
||||
// this phase at all: a skill with no machine-checkable consequence
|
||||
// here is one an agent is right to pass over, and scoring that as a
|
||||
// failure would punish correct triage.
|
||||
// Both retrieval arms: offered by name and `when_to_use`, and never
|
||||
// opened. Whether that is a miss depends on whether the skill had
|
||||
// anything to say about this phase at all: a skill with no
|
||||
// machine-checkable consequence here is one an agent is right to pass
|
||||
// over, and scoring that as a failure would punish correct triage.
|
||||
crate::skill_delivery::Mode::Index | crate::skill_delivery::Mode::Files => {
|
||||
if matches!(compliance, Verdict::NotApplicable)
|
||||
&& matches!(boundary, Verdict::NotApplicable)
|
||||
{
|
||||
@@ -948,6 +961,92 @@ mod tests {
|
||||
}
|
||||
}
|
||||
|
||||
fn rendered_files(skills: &[(&str, &str)]) -> String {
|
||||
let body: String = skills
|
||||
.iter()
|
||||
.map(|(name, when)| {
|
||||
crate::topology_exec::render_pinned_skill(
|
||||
name,
|
||||
&crate::skill_delivery::file_entry(
|
||||
"a procedure",
|
||||
Some(when),
|
||||
&crate::skill_delivery::skill_file_path(name),
|
||||
),
|
||||
)
|
||||
})
|
||||
.collect();
|
||||
format!(
|
||||
"Task: x\n\n# Your skills\n\n{}\n{body}",
|
||||
crate::skill_delivery::FILES_PREAMBLE
|
||||
)
|
||||
}
|
||||
|
||||
fn read_file(path: &str) -> ToolEvidence {
|
||||
ToolEvidence {
|
||||
tool: "Read".into(),
|
||||
path: Some(path.into()),
|
||||
input: json!({ "file_path": path }),
|
||||
response: serde_json::Value::Null,
|
||||
}
|
||||
}
|
||||
|
||||
/// The `files` arm's loop, end to end, for the same reason as the uri
|
||||
/// test below: `skill_file_path` writes the path, `file_entry` puts it in
|
||||
/// the prompt, `skill_from_file_path` reads it back off a `Read`.
|
||||
#[test]
|
||||
fn the_path_the_files_arm_advertises_is_the_one_the_scorer_recovers() {
|
||||
let prompt = rendered_files(&[("workspace-repo-commit-protocol", "before committing")]);
|
||||
assert!(
|
||||
prompt.contains("Read(file_path=\"/mission/skills/workspace-repo-commit-protocol.md\")"),
|
||||
"the entry must name the file to read:\n{prompt}"
|
||||
);
|
||||
assert_eq!(
|
||||
crate::skill_delivery::mode_in_prompt(&prompt),
|
||||
crate::skill_delivery::Mode::Files
|
||||
);
|
||||
let tools = vec![read_file("/mission/skills/workspace-repo-commit-protocol.md")];
|
||||
let ev = Evidence::new("", &tools);
|
||||
assert_eq!(retrieved_skills(&ev), vec!["workspace-repo-commit-protocol"]);
|
||||
let scored = score(&prompt, &ev, &builtin);
|
||||
assert_eq!(scored.len(), 1);
|
||||
assert!(
|
||||
matches!(scored[0].trigger, Verdict::Pass),
|
||||
"a Read of the advertised path IS the Trigger axis: {:?}",
|
||||
scored[0].trigger
|
||||
);
|
||||
}
|
||||
|
||||
/// Agents read files constantly. Only a `Read` INSIDE the skills directory
|
||||
/// is a retrieval; anything else scoring as one would make Trigger a count
|
||||
/// of file reads.
|
||||
#[test]
|
||||
fn an_ordinary_read_is_not_a_retrieval() {
|
||||
let tools = vec![
|
||||
read_file("/mission/repo/research/REPORT.md"),
|
||||
read_file("/mission/skills"),
|
||||
read_file("/mission/skills/nested/x.md"),
|
||||
read_file("/etc/passwd"),
|
||||
];
|
||||
assert!(retrieved_skills(&Evidence::new("", &tools)).is_empty());
|
||||
}
|
||||
|
||||
/// Under `files`, never opened is a miss when the skill had a checkable
|
||||
/// consequence — the same rule as `index`, and NOT `inline`'s
|
||||
/// "not observable", which would report the arm's own defect as a blind spot.
|
||||
#[test]
|
||||
fn the_files_arm_scores_a_miss_like_the_index_arm() {
|
||||
let prompt = rendered_files(&[("int-xx-marker-protocol", "when writing markers")]);
|
||||
let tools: Vec<ToolEvidence> = vec![];
|
||||
let ev = Evidence::new("INT-01 something without the required shape", &tools);
|
||||
let scored = score(&prompt, &ev, &builtin);
|
||||
assert_eq!(scored.len(), 1);
|
||||
assert!(
|
||||
!matches!(scored[0].trigger, Verdict::NotObservable(_)),
|
||||
"files is a retrieval arm; a miss must not read as inline: {:?}",
|
||||
scored[0].trigger
|
||||
);
|
||||
}
|
||||
|
||||
/// The whole loop, end to end: the index writes a uri, the agent reads that
|
||||
/// exact uri back, and the scorer recovers the skill's name from it.
|
||||
///
|
||||
|
||||
Reference in New Issue
Block a user