feat(skills): a files arm — progressive disclosure through Read, not a deferred tool
deploy / test (push) Successful in 5m18s
deploy / build (push) Successful in 5m36s

The `index` arm retrieves through `ReadMcpResourceTool`, which is DEFERRED:
absent from the agent's default tool list until `ToolSearch` loads it. Across
three matched production runs (same recipe, same task, same three offered
uris) it retrieved 1 skill in 9 chances:

    01a07812  delegation forced      no instruction    0/3
    01a0842e  no delegation          no instruction    1/3
    01a09877  no delegation          told to load it   0/3

The third run is the decisive one. The preamble said in plain words to run
ToolSearch first; all three prompts carried it; zero ToolSearch calls, and the
three reasoning narratives never mention skills at all. The section was not
declined, it was never engaged with. Instruction is not the lever.

`Read` is a core tool. Never deferred, and every one of those agents used it.
So this arm keeps progressive disclosure exactly as `index` has it — a name, a
`when_to_use`, and a pointer the agent has to follow — and changes only what
the pointer is: a path under /mission/skills instead of an MCP uri. The bodies
are written into the container at launch (every visible skill, one tar upload;
bindings resolve per agent at turn time so a per-mission subset is not knowable
here) and a `Read` of that path is a tapped tool call, so Trigger is exactly as
observable as before.

A third arm and not a replacement, selected per mission like the others, so
the comparison runs against one binary. `resolve` falls back to `inline` when
the files were not written, for the reason `index` does: a pointer to nothing
reads as an agent ignoring its skills.

The writer and reader of a path are one pair of functions
(`skill_file_path` / `skill_from_file_path`), matched by the scorer through
the same seam `parse_uri` uses, and the end-to-end test fails when the matcher
is broken. `Mode::is_retrieval` exists so the next arm cannot silently inherit
`inline`'s "not observable" for what is a miss.

Co-Authored-By: Claude Opus 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01WZb5A2kfVfjpdwSochkuHz
This commit is contained in:
Omar Sobh
2026-09-12 22:33:41 -05:00
co-authored by Claude Opus 5
parent 8d6310f126
commit 00160739de
7 changed files with 421 additions and 61 deletions
+111 -12
View File
@@ -174,13 +174,26 @@ impl<'a> Evidence<'a> {
pub fn retrieved_skills(ev: &Evidence<'_>) -> Vec<String> {
let mut out = Vec::new();
for t in ev.tools {
if t.tool != "ReadMcpResourceTool" {
continue;
}
let Some(uri) = t.input.get("uri").and_then(|v| v.as_str()) else {
continue;
let name = match t.tool.as_str() {
"ReadMcpResourceTool" => t
.input
.get("uri")
.and_then(|v| v.as_str())
.and_then(crate::mcp_skills::parse_uri)
.map(|(_, name)| name),
// The `files` arm: the pointer is a path and the retrieval is a
// plain `Read`. Matched through `skill_from_file_path`, the reader
// half of the function that wrote the path, for the same reason
// the uri goes through `parse_uri`. A `Read` anywhere else is an
// ordinary file read and is not a retrieval of anything.
"Read" => t
.input
.get("file_path")
.and_then(|v| v.as_str())
.and_then(crate::skill_delivery::skill_from_file_path),
_ => None,
};
if let Some((_, name)) = crate::mcp_skills::parse_uri(uri) {
if let Some(name) = name {
if !out.contains(&name) {
out.push(name);
}
@@ -216,12 +229,12 @@ fn trigger_verdict(
through the door instead and this becomes a tool call"
.into(),
),
crate::skill_delivery::Mode::Index => {
// Offered by name and `when_to_use`, and never opened. Whether that
// is a miss depends on whether the skill had anything to say about
// this phase at all: a skill with no machine-checkable consequence
// here is one an agent is right to pass over, and scoring that as a
// failure would punish correct triage.
// Both retrieval arms: offered by name and `when_to_use`, and never
// opened. Whether that is a miss depends on whether the skill had
// anything to say about this phase at all: a skill with no
// machine-checkable consequence here is one an agent is right to pass
// over, and scoring that as a failure would punish correct triage.
crate::skill_delivery::Mode::Index | crate::skill_delivery::Mode::Files => {
if matches!(compliance, Verdict::NotApplicable)
&& matches!(boundary, Verdict::NotApplicable)
{
@@ -948,6 +961,92 @@ mod tests {
}
}
fn rendered_files(skills: &[(&str, &str)]) -> String {
let body: String = skills
.iter()
.map(|(name, when)| {
crate::topology_exec::render_pinned_skill(
name,
&crate::skill_delivery::file_entry(
"a procedure",
Some(when),
&crate::skill_delivery::skill_file_path(name),
),
)
})
.collect();
format!(
"Task: x\n\n# Your skills\n\n{}\n{body}",
crate::skill_delivery::FILES_PREAMBLE
)
}
fn read_file(path: &str) -> ToolEvidence {
ToolEvidence {
tool: "Read".into(),
path: Some(path.into()),
input: json!({ "file_path": path }),
response: serde_json::Value::Null,
}
}
/// The `files` arm's loop, end to end, for the same reason as the uri
/// test below: `skill_file_path` writes the path, `file_entry` puts it in
/// the prompt, `skill_from_file_path` reads it back off a `Read`.
#[test]
fn the_path_the_files_arm_advertises_is_the_one_the_scorer_recovers() {
let prompt = rendered_files(&[("workspace-repo-commit-protocol", "before committing")]);
assert!(
prompt.contains("Read(file_path=\"/mission/skills/workspace-repo-commit-protocol.md\")"),
"the entry must name the file to read:\n{prompt}"
);
assert_eq!(
crate::skill_delivery::mode_in_prompt(&prompt),
crate::skill_delivery::Mode::Files
);
let tools = vec![read_file("/mission/skills/workspace-repo-commit-protocol.md")];
let ev = Evidence::new("", &tools);
assert_eq!(retrieved_skills(&ev), vec!["workspace-repo-commit-protocol"]);
let scored = score(&prompt, &ev, &builtin);
assert_eq!(scored.len(), 1);
assert!(
matches!(scored[0].trigger, Verdict::Pass),
"a Read of the advertised path IS the Trigger axis: {:?}",
scored[0].trigger
);
}
/// Agents read files constantly. Only a `Read` INSIDE the skills directory
/// is a retrieval; anything else scoring as one would make Trigger a count
/// of file reads.
#[test]
fn an_ordinary_read_is_not_a_retrieval() {
let tools = vec![
read_file("/mission/repo/research/REPORT.md"),
read_file("/mission/skills"),
read_file("/mission/skills/nested/x.md"),
read_file("/etc/passwd"),
];
assert!(retrieved_skills(&Evidence::new("", &tools)).is_empty());
}
/// Under `files`, never opened is a miss when the skill had a checkable
/// consequence — the same rule as `index`, and NOT `inline`'s
/// "not observable", which would report the arm's own defect as a blind spot.
#[test]
fn the_files_arm_scores_a_miss_like_the_index_arm() {
let prompt = rendered_files(&[("int-xx-marker-protocol", "when writing markers")]);
let tools: Vec<ToolEvidence> = vec![];
let ev = Evidence::new("INT-01 something without the required shape", &tools);
let scored = score(&prompt, &ev, &builtin);
assert_eq!(scored.len(), 1);
assert!(
!matches!(scored[0].trigger, Verdict::NotObservable(_)),
"files is a retrieval arm; a miss must not read as inline: {:?}",
scored[0].trigger
);
}
/// The whole loop, end to end: the index writes a uri, the agent reads that
/// exact uri back, and the scorer recovers the skill's name from it.
///