fix(skills): the door is a deferred tool, so say how to load it
`READ_IT` has always named `ReadMcpResourceTool` in every index entry. That is
not enough, because the tool is DEFERRED — not on the agent's default list, and
uncallable until `ToolSearch` loads its schema. Naming a tool the agent cannot
call reads, from the outside, exactly like an agent ignoring its skills.
Measured on a matched pair in production. Same recipe, same `index` arm, same
three offered uris, one variable:
01a07812 76 tool calls, ToolSearch x4 (web_fetch, RemoteTrigger),
never searched for the door -> 0 skills retrieved
01a0842e ToolSearch(select:ReadMcpResourceTool), then the fetch
-> 1 skill retrieved, trigger=pass
One agent worked the extra step out unprompted; the other did not. A capability
that depends on the model guessing a tool is loadable is not delivered, so the
preamble now says the step out loud.
The reader keeps both spellings. `mode_in_prompt` scores the arm off a RECORDED
prompt and `retain_events_until` holds those for 90 days, so editing the writer
alone would have re-labelled every stored `index` run as `inline` — including
the pair above, whose whole value is that they are comparable. `INDEX_PREAMBLE_V1`
is kept as a reader-only constant and matched alongside the current text.
Verified rather than assumed: the real stored prompt from `01a07812` still
matches V1 as an exact line, the compatibility test fails when the fallback is
removed, and a second test asserts V1 stays a prefix of the current preamble
since `concat!` cannot take a const.
Co-Authored-By: Claude Opus 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01WZb5A2kfVfjpdwSochkuHz
This commit is contained in:
co-authored by
Claude Opus 5
parent
42c24de6a9
commit
1072964326
@@ -53,15 +53,42 @@ pub const ENV_VAR: &str = "CLAWMATES_SKILL_DELIVERY";
|
||||
pub const INLINE_PREAMBLE: &str = "These are procedures you are expected to follow for \
|
||||
this kind of work. Where one applies to what you are about to do, follow it.";
|
||||
|
||||
/// The `# Your skills` preamble under [`Mode::Index`] as first shipped.
|
||||
///
|
||||
/// Kept because [`mode_in_prompt`] reads the arm off a RECORDED prompt, and
|
||||
/// prompts composed before the tool-loading sentence was added are still being
|
||||
/// scored — `retain_events_until` holds them for 90 days. Dropping this
|
||||
/// constant would silently re-label every stored `index` run as `inline` and
|
||||
/// report Trigger against the wrong arm.
|
||||
///
|
||||
/// Never send this one. It is a reader, not a writer.
|
||||
pub const INDEX_PREAMBLE_V1: &str = "These procedures are AVAILABLE to you; their bodies are \
|
||||
not included below. Each entry names one, says when it applies, and gives the uri that \
|
||||
returns it. Where an entry applies to what you are about to do, read it FIRST and then \
|
||||
follow it.";
|
||||
|
||||
/// The `# Your skills` preamble under [`Mode::Index`].
|
||||
///
|
||||
/// Written and matched in one place ([`mode_in_prompt`]) so the reader cannot
|
||||
/// drift from the writer — the same rule `SKILL_MARKER` is under, and for the
|
||||
/// same reason: a scorer that misreads the arm reports the wrong axis.
|
||||
///
|
||||
/// # Why the last sentence exists
|
||||
///
|
||||
/// `ReadMcpResourceTool` is a DEFERRED tool: it is not on the agent's default
|
||||
/// tool list and cannot be called until `ToolSearch` loads its schema. Naming
|
||||
/// it — which [`READ_IT`] already did — is therefore not enough, and the
|
||||
/// difference is measurable. Prod mission `01a07812` made 76 tool calls,
|
||||
/// searched for two other tools, never searched for this one, and retrieved
|
||||
/// ZERO skills. `01a0842e`, same recipe and same offered uris, ran
|
||||
/// `ToolSearch(select:ReadMcpResourceTool)` and then fetched. One agent worked
|
||||
/// the extra step out on its own; the other did not, and a capability that
|
||||
/// depends on the model guessing that a tool is loadable is not delivered.
|
||||
pub const INDEX_PREAMBLE: &str = "These procedures are AVAILABLE to you; their bodies are \
|
||||
not included below. Each entry names one, says when it applies, and gives the uri that \
|
||||
returns it. Where an entry applies to what you are about to do, read it FIRST and then \
|
||||
follow it.";
|
||||
follow it. ReadMcpResourceTool may not be loaded in this session: if you do not already \
|
||||
have it, run ToolSearch with the query select:ReadMcpResourceTool before your first read.";
|
||||
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
pub enum Mode {
|
||||
@@ -166,7 +193,13 @@ pub fn preamble(mode: Mode) -> &'static str {
|
||||
/// Matched as a whole line. A skill body that quotes the preamble mid-sentence
|
||||
/// is prose; this is the same rule `skill_names_in` learned the hard way.
|
||||
pub fn mode_in_prompt(prompt: &str) -> Mode {
|
||||
if prompt.lines().any(|l| l.trim() == INDEX_PREAMBLE) {
|
||||
// Both spellings, because this reads prompts composed by older builds as
|
||||
// well as the current one. A stored measurement that changes arm when the
|
||||
// writer is edited is not a measurement.
|
||||
if prompt
|
||||
.lines()
|
||||
.any(|l| l.trim() == INDEX_PREAMBLE || l.trim() == INDEX_PREAMBLE_V1)
|
||||
{
|
||||
Mode::Index
|
||||
} else {
|
||||
Mode::Inline
|
||||
@@ -265,6 +298,31 @@ mod tests {
|
||||
assert_eq!(resolve(Mode::Inline, true), Mode::Inline);
|
||||
}
|
||||
|
||||
/// A prompt composed before the tool-loading sentence existed must still
|
||||
/// score as `Index`. Stored prompts are held for 90 days and re-scored
|
||||
/// when the scorer changes; if this regressed, every one of them would
|
||||
/// quietly become an `inline` run and Trigger would be reported against an
|
||||
/// arm that never ran.
|
||||
#[test]
|
||||
fn an_older_index_prompt_still_reads_as_index() {
|
||||
let old = format!("Task: x\n\n# Your skills\n\n{INDEX_PREAMBLE_V1}\n\nentry");
|
||||
assert_eq!(mode_in_prompt(&old), Mode::Index);
|
||||
let new = format!("Task: x\n\n# Your skills\n\n{INDEX_PREAMBLE}\n\nentry");
|
||||
assert_eq!(mode_in_prompt(&new), Mode::Index);
|
||||
}
|
||||
|
||||
/// The two spellings must stay one text plus an addition, not two texts.
|
||||
/// Written out in full because `concat!` cannot take a const, so nothing
|
||||
/// but this test stops them drifting apart.
|
||||
#[test]
|
||||
fn the_current_preamble_extends_the_original() {
|
||||
assert!(
|
||||
INDEX_PREAMBLE.starts_with(INDEX_PREAMBLE_V1),
|
||||
"the v1 preamble must remain a prefix, or old prompts stop matching"
|
||||
);
|
||||
assert!(INDEX_PREAMBLE.contains("select:ReadMcpResourceTool"));
|
||||
}
|
||||
|
||||
/// The scorer reads the arm off the prompt, so the writer and this reader
|
||||
/// have to agree for every arm — including the one that writes no marker.
|
||||
#[test]
|
||||
|
||||
Reference in New Issue
Block a user