fix(skills): the door is a deferred tool, so say how to load it
`READ_IT` has always named `ReadMcpResourceTool` in every index entry. That is
not enough, because the tool is DEFERRED — not on the agent's default list, and
uncallable until `ToolSearch` loads its schema. Naming a tool the agent cannot
call reads, from the outside, exactly like an agent ignoring its skills.
Measured on a matched pair in production. Same recipe, same `index` arm, same
three offered uris, one variable:
01a07812 76 tool calls, ToolSearch x4 (web_fetch, RemoteTrigger),
never searched for the door -> 0 skills retrieved
01a0842e ToolSearch(select:ReadMcpResourceTool), then the fetch
-> 1 skill retrieved, trigger=pass
One agent worked the extra step out unprompted; the other did not. A capability
that depends on the model guessing a tool is loadable is not delivered, so the
preamble now says the step out loud.
The reader keeps both spellings. `mode_in_prompt` scores the arm off a RECORDED
prompt and `retain_events_until` holds those for 90 days, so editing the writer
alone would have re-labelled every stored `index` run as `inline` — including
the pair above, whose whole value is that they are comparable. `INDEX_PREAMBLE_V1`
is kept as a reader-only constant and matched alongside the current text.
Verified rather than assumed: the real stored prompt from `01a07812` still
matches V1 as an exact line, the compatibility test fails when the fallback is
removed, and a second test asserts V1 stays a prefix of the current preamble
since `concat!` cannot take a const.
Co-Authored-By: Claude Opus 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01WZb5A2kfVfjpdwSochkuHz
This commit is contained in:
co-authored by
Claude Opus 5
parent
42c24de6a9
commit
1072964326
@@ -53,15 +53,42 @@ pub const ENV_VAR: &str = "CLAWMATES_SKILL_DELIVERY";
|
|||||||
pub const INLINE_PREAMBLE: &str = "These are procedures you are expected to follow for \
|
pub const INLINE_PREAMBLE: &str = "These are procedures you are expected to follow for \
|
||||||
this kind of work. Where one applies to what you are about to do, follow it.";
|
this kind of work. Where one applies to what you are about to do, follow it.";
|
||||||
|
|
||||||
|
/// The `# Your skills` preamble under [`Mode::Index`] as first shipped.
|
||||||
|
///
|
||||||
|
/// Kept because [`mode_in_prompt`] reads the arm off a RECORDED prompt, and
|
||||||
|
/// prompts composed before the tool-loading sentence was added are still being
|
||||||
|
/// scored — `retain_events_until` holds them for 90 days. Dropping this
|
||||||
|
/// constant would silently re-label every stored `index` run as `inline` and
|
||||||
|
/// report Trigger against the wrong arm.
|
||||||
|
///
|
||||||
|
/// Never send this one. It is a reader, not a writer.
|
||||||
|
pub const INDEX_PREAMBLE_V1: &str = "These procedures are AVAILABLE to you; their bodies are \
|
||||||
|
not included below. Each entry names one, says when it applies, and gives the uri that \
|
||||||
|
returns it. Where an entry applies to what you are about to do, read it FIRST and then \
|
||||||
|
follow it.";
|
||||||
|
|
||||||
/// The `# Your skills` preamble under [`Mode::Index`].
|
/// The `# Your skills` preamble under [`Mode::Index`].
|
||||||
///
|
///
|
||||||
/// Written and matched in one place ([`mode_in_prompt`]) so the reader cannot
|
/// Written and matched in one place ([`mode_in_prompt`]) so the reader cannot
|
||||||
/// drift from the writer — the same rule `SKILL_MARKER` is under, and for the
|
/// drift from the writer — the same rule `SKILL_MARKER` is under, and for the
|
||||||
/// same reason: a scorer that misreads the arm reports the wrong axis.
|
/// same reason: a scorer that misreads the arm reports the wrong axis.
|
||||||
|
///
|
||||||
|
/// # Why the last sentence exists
|
||||||
|
///
|
||||||
|
/// `ReadMcpResourceTool` is a DEFERRED tool: it is not on the agent's default
|
||||||
|
/// tool list and cannot be called until `ToolSearch` loads its schema. Naming
|
||||||
|
/// it — which [`READ_IT`] already did — is therefore not enough, and the
|
||||||
|
/// difference is measurable. Prod mission `01a07812` made 76 tool calls,
|
||||||
|
/// searched for two other tools, never searched for this one, and retrieved
|
||||||
|
/// ZERO skills. `01a0842e`, same recipe and same offered uris, ran
|
||||||
|
/// `ToolSearch(select:ReadMcpResourceTool)` and then fetched. One agent worked
|
||||||
|
/// the extra step out on its own; the other did not, and a capability that
|
||||||
|
/// depends on the model guessing that a tool is loadable is not delivered.
|
||||||
pub const INDEX_PREAMBLE: &str = "These procedures are AVAILABLE to you; their bodies are \
|
pub const INDEX_PREAMBLE: &str = "These procedures are AVAILABLE to you; their bodies are \
|
||||||
not included below. Each entry names one, says when it applies, and gives the uri that \
|
not included below. Each entry names one, says when it applies, and gives the uri that \
|
||||||
returns it. Where an entry applies to what you are about to do, read it FIRST and then \
|
returns it. Where an entry applies to what you are about to do, read it FIRST and then \
|
||||||
follow it.";
|
follow it. ReadMcpResourceTool may not be loaded in this session: if you do not already \
|
||||||
|
have it, run ToolSearch with the query select:ReadMcpResourceTool before your first read.";
|
||||||
|
|
||||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||||
pub enum Mode {
|
pub enum Mode {
|
||||||
@@ -166,7 +193,13 @@ pub fn preamble(mode: Mode) -> &'static str {
|
|||||||
/// Matched as a whole line. A skill body that quotes the preamble mid-sentence
|
/// Matched as a whole line. A skill body that quotes the preamble mid-sentence
|
||||||
/// is prose; this is the same rule `skill_names_in` learned the hard way.
|
/// is prose; this is the same rule `skill_names_in` learned the hard way.
|
||||||
pub fn mode_in_prompt(prompt: &str) -> Mode {
|
pub fn mode_in_prompt(prompt: &str) -> Mode {
|
||||||
if prompt.lines().any(|l| l.trim() == INDEX_PREAMBLE) {
|
// Both spellings, because this reads prompts composed by older builds as
|
||||||
|
// well as the current one. A stored measurement that changes arm when the
|
||||||
|
// writer is edited is not a measurement.
|
||||||
|
if prompt
|
||||||
|
.lines()
|
||||||
|
.any(|l| l.trim() == INDEX_PREAMBLE || l.trim() == INDEX_PREAMBLE_V1)
|
||||||
|
{
|
||||||
Mode::Index
|
Mode::Index
|
||||||
} else {
|
} else {
|
||||||
Mode::Inline
|
Mode::Inline
|
||||||
@@ -265,6 +298,31 @@ mod tests {
|
|||||||
assert_eq!(resolve(Mode::Inline, true), Mode::Inline);
|
assert_eq!(resolve(Mode::Inline, true), Mode::Inline);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// A prompt composed before the tool-loading sentence existed must still
|
||||||
|
/// score as `Index`. Stored prompts are held for 90 days and re-scored
|
||||||
|
/// when the scorer changes; if this regressed, every one of them would
|
||||||
|
/// quietly become an `inline` run and Trigger would be reported against an
|
||||||
|
/// arm that never ran.
|
||||||
|
#[test]
|
||||||
|
fn an_older_index_prompt_still_reads_as_index() {
|
||||||
|
let old = format!("Task: x\n\n# Your skills\n\n{INDEX_PREAMBLE_V1}\n\nentry");
|
||||||
|
assert_eq!(mode_in_prompt(&old), Mode::Index);
|
||||||
|
let new = format!("Task: x\n\n# Your skills\n\n{INDEX_PREAMBLE}\n\nentry");
|
||||||
|
assert_eq!(mode_in_prompt(&new), Mode::Index);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The two spellings must stay one text plus an addition, not two texts.
|
||||||
|
/// Written out in full because `concat!` cannot take a const, so nothing
|
||||||
|
/// but this test stops them drifting apart.
|
||||||
|
#[test]
|
||||||
|
fn the_current_preamble_extends_the_original() {
|
||||||
|
assert!(
|
||||||
|
INDEX_PREAMBLE.starts_with(INDEX_PREAMBLE_V1),
|
||||||
|
"the v1 preamble must remain a prefix, or old prompts stop matching"
|
||||||
|
);
|
||||||
|
assert!(INDEX_PREAMBLE.contains("select:ReadMcpResourceTool"));
|
||||||
|
}
|
||||||
|
|
||||||
/// The scorer reads the arm off the prompt, so the writer and this reader
|
/// The scorer reads the arm off the prompt, so the writer and this reader
|
||||||
/// have to agree for every arm — including the one that writes no marker.
|
/// have to agree for every arm — including the one that writes no marker.
|
||||||
#[test]
|
#[test]
|
||||||
|
|||||||
Reference in New Issue
Block a user