feat(skill-use): a compliance check for web-search-triage, from what the tap recorded
The scorer could see that agents OPENED web-search-triage (trigger=pass on both files-arm runs) and nothing about whether they followed it — compliance was not_applicable because no mechanical check existed. The evidence was in the recorded arguments the whole time. On the runs that read the skill, the parent decomposed the sweep into per-source fetches and sent each to a subagent; on 01a09b42 two of those spawn prompts read "Return the URL, date if visible, and the key content". The task never asked for a date. The skill's "undated is a finding" did. On the runs that did not read it: inline curls, no subagents, no date. Two of the skill's rules leave a mark in arguments, and the check scores exactly those two. The ranking rule: every URL a fetch was sent to is classified against a short allow-list of primary hosts (rank 0) and a short skip-list of aggregators (rank 3+); fetching an aggregator is the visible violation, fetching primary sources the visible compliance, and anything unrecognised is unranked and decides nothing. The date rule: reported as extra evidence on a pass, never required for one, because a curl to an abstract page has no prompt to ask in. `Agent` is a fetching tool here on purpose. The URLs on the files-arm runs live in the spawn PROMPT; a check that only read curl lines would have scored those runs as fetching nothing. `Verdict::PassWith(String)` carries the evidence and serialises under the same "pass" tag, so no reader grows a fourth branch and the one that looks finds the date fingerprint in `why`. One-sided like every check in this module: no tools is not observable, no fetch is not applicable, an unrankable fetch is not a violation. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01WZb5A2kfVfjpdwSochkuHz
This commit is contained in:
Omar Sobh
co-authored by
Claude Opus 5
Claude Opus 5
parent
7dd3aa0965
commit
758760cedb
@@ -55,6 +55,15 @@ use serde::Serialize;
|
||||
#[serde(rename_all = "snake_case", tag = "verdict", content = "why")]
|
||||
pub enum Verdict {
|
||||
Pass,
|
||||
/// A pass that can say what it saw. Same tag as [`Verdict::Pass`] on the
|
||||
/// wire — `{"verdict":"pass","why":…}` — so every reader that keys on the
|
||||
/// tag is unaffected and the evidence is there for the one that looks.
|
||||
///
|
||||
/// Exists because a bare pass on `web-search-triage` would have hidden
|
||||
/// the only finding worth having: the agent's spawn prompts asked for the
|
||||
/// page's date, which the task never did and the skill does.
|
||||
#[serde(rename = "pass")]
|
||||
PassWith(String),
|
||||
Fail(String),
|
||||
/// The skill says nothing this axis can check.
|
||||
NotApplicable,
|
||||
@@ -68,7 +77,7 @@ pub enum Verdict {
|
||||
impl Verdict {
|
||||
pub fn label(&self) -> &'static str {
|
||||
match self {
|
||||
Verdict::Pass => "pass",
|
||||
Verdict::Pass | Verdict::PassWith(_) => "pass",
|
||||
Verdict::Fail(_) => "FAIL",
|
||||
Verdict::NotApplicable => "n/a",
|
||||
Verdict::NotObservable(_) => "not observable",
|
||||
@@ -313,6 +322,7 @@ fn check(skill: &str, ev: &Evidence<'_>) -> (Verdict, Verdict) {
|
||||
"arxiv-daily" => (Verdict::NotApplicable, arxiv_boundary(ev)),
|
||||
"workspace-repo-commit-protocol" => (Verdict::NotApplicable, workspace_boundary(ev)),
|
||||
"small-focused-commits" => (Verdict::NotApplicable, commit_subject_boundary(ev)),
|
||||
"web-search-triage" => (triage_compliance(ev), Verdict::NotApplicable),
|
||||
// One check for both: `cargo-test-driven-development` is the Rust
|
||||
// flavour of the same loop, and its own text says so. Scoring them by
|
||||
// separate rules would mean two rules for one procedure, which is how
|
||||
@@ -775,6 +785,164 @@ fn dash_m_value(cmd: &str) -> Option<String> {
|
||||
None
|
||||
}
|
||||
|
||||
/// The tools whose arguments carry a URL the agent reached for.
|
||||
///
|
||||
/// `Agent` is here on purpose. On both `files`-arm runs the parent decomposed
|
||||
/// the sweep into per-source fetches and sent each to a subagent — the URLs
|
||||
/// live in the spawn PROMPT, and a check that only read `curl` lines would
|
||||
/// have scored those runs as fetching nothing.
|
||||
const FETCH_TOOLS: &[&str] = &["Bash", "Agent", "WebFetch", "web_fetch"];
|
||||
|
||||
/// Every `http(s)` URL in the arguments of a fetching tool, in call order.
|
||||
fn fetched_urls(ev: &Evidence<'_>) -> Vec<String> {
|
||||
let mut out = Vec::new();
|
||||
for t in ev.tools {
|
||||
if !FETCH_TOOLS.contains(&t.tool.as_str()) {
|
||||
continue;
|
||||
}
|
||||
// A `Bash` that never fetches is most of what agents run.
|
||||
if t.tool == "Bash"
|
||||
&& !t
|
||||
.command()
|
||||
.is_some_and(|c| c.contains("curl") || c.contains("wget"))
|
||||
{
|
||||
continue;
|
||||
}
|
||||
let text = t.input.to_string();
|
||||
let mut rest = text.as_str();
|
||||
while let Some(i) = rest.find("http") {
|
||||
let cand = &rest[i..];
|
||||
let end = cand
|
||||
.find(|c: char| {
|
||||
c.is_whitespace() || matches!(c, '"' | '\'' | '<' | '>' | ')' | ']' | '\\')
|
||||
})
|
||||
.unwrap_or(cand.len());
|
||||
let url = cand[..end].trim_end_matches(|c| matches!(c, '.' | ',' | ';' | ':'));
|
||||
if url.starts_with("http://") || url.starts_with("https://") {
|
||||
out.push(url.to_string());
|
||||
}
|
||||
rest = &cand[end.max(4)..];
|
||||
}
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
/// Rank 0 in `web-search-triage`'s ladder: the paper, the spec, the release.
|
||||
///
|
||||
/// A conservative allow-list. Anything not on it is UNRANKED, not rank 3 — a
|
||||
/// vendor's docs, an author's own blog and a lab page are all rank 0 or 1 and
|
||||
/// none of them can be recognised by hostname.
|
||||
fn is_primary_source(url: &str) -> bool {
|
||||
const HOSTS: &[&str] = &[
|
||||
"arxiv.org/abs/",
|
||||
"arxiv.org/pdf/",
|
||||
"doi.org/",
|
||||
"aclanthology.org/",
|
||||
"openreview.net/",
|
||||
"proceedings.neurips.cc/",
|
||||
"proceedings.mlr.press/",
|
||||
"dl.acm.org/doi/",
|
||||
"ieeexplore.ieee.org/",
|
||||
"github.com/",
|
||||
"nature.com/articles/",
|
||||
"science.org/doi/",
|
||||
];
|
||||
HOSTS.iter().any(|h| url.contains(h))
|
||||
}
|
||||
|
||||
/// Rank 3 and below: a restatement of a restatement. The skill's own words are
|
||||
/// "the same item and should be recorded once, if at all", and its skip signals
|
||||
/// — "a numbered list of tools", no date — describe these hosts.
|
||||
///
|
||||
/// Also conservative. Substack, X and personal blogs are NOT here: an author's
|
||||
/// own post is rank 1 and the skill says to read it.
|
||||
fn is_aggregator(url: &str) -> bool {
|
||||
const HOSTS: &[&str] = &[
|
||||
"medium.com/",
|
||||
"towardsdatascience.com/",
|
||||
"reddit.com/",
|
||||
"news.ycombinator.com/",
|
||||
"quora.com/",
|
||||
"dev.to/",
|
||||
"linkedin.com/",
|
||||
"wikipedia.org/",
|
||||
];
|
||||
HOSTS.iter().any(|h| url.contains(h))
|
||||
}
|
||||
|
||||
/// `web-search-triage`: read the primary source, and treat undated as a finding.
|
||||
///
|
||||
/// Two of the skill's rules leave a mark in the recorded ARGUMENTS, and this
|
||||
/// scores exactly those two — the rest of the skill is judgement about page
|
||||
/// content the tap never sees, and a heuristic over it would be a number that
|
||||
/// looks like a measurement and is not one.
|
||||
///
|
||||
/// - The ranking rule. Every URL a fetch was sent to is classified against a
|
||||
/// short allow-list of primary hosts and a short skip-list of aggregators.
|
||||
/// Fetching an aggregator is the visible violation; fetching primary sources
|
||||
/// is the visible compliance. Anything unrecognised is unranked and decides
|
||||
/// nothing.
|
||||
/// - The date rule. On mission `01a09b42` the parent's spawn prompts read
|
||||
/// "Return the URL, date if visible, and the key content" — the task never
|
||||
/// asked for a date; the skill's "undated is a finding" did. Reported as
|
||||
/// extra evidence on a pass, never required for one: a curl to an abstract
|
||||
/// page has no prompt to ask in.
|
||||
///
|
||||
/// One-sided like every check here: it reports a violation it can see and never
|
||||
/// infers compliance from silence.
|
||||
fn triage_compliance(ev: &Evidence<'_>) -> Verdict {
|
||||
if ev.tools.is_empty() {
|
||||
return Verdict::NotObservable(
|
||||
"no tool calls were recorded for this mission — which URLs were \
|
||||
fetched is what this check reads, and that is not in the narrative"
|
||||
.into(),
|
||||
);
|
||||
}
|
||||
let urls = fetched_urls(ev);
|
||||
if urls.is_empty() {
|
||||
// Never swept the web, so there was nothing to triage.
|
||||
return Verdict::NotApplicable;
|
||||
}
|
||||
if let Some(u) = urls.iter().find(|u| is_aggregator(u)) {
|
||||
return Verdict::Fail(format!(
|
||||
"fetched {u} — an aggregator, rank 3 or below on the skill's ladder; \
|
||||
the procedure is to find the primary source and record the rest as \
|
||||
one item, not to read them"
|
||||
));
|
||||
}
|
||||
let primary = urls.iter().filter(|u| is_primary_source(u)).count();
|
||||
if primary == 0 {
|
||||
return Verdict::NotObservable(format!(
|
||||
"fetched {} URL(s), none on the primary-source list and none on the \
|
||||
aggregator list — the check cannot rank them, and a rank it cannot \
|
||||
see is not a violation",
|
||||
urls.len()
|
||||
));
|
||||
}
|
||||
let asked_for_date = ev
|
||||
.tools
|
||||
.iter()
|
||||
.filter(|t| t.tool == "Agent")
|
||||
.filter(|t| {
|
||||
t.input
|
||||
.get("prompt")
|
||||
.and_then(serde_json::Value::as_str)
|
||||
.is_some_and(|p| p.to_ascii_lowercase().contains("date"))
|
||||
})
|
||||
.count();
|
||||
let mut why = format!(
|
||||
"fetched {primary} primary source(s) out of {} URL(s) and no aggregator",
|
||||
urls.len()
|
||||
);
|
||||
if asked_for_date > 0 {
|
||||
why.push_str(&format!(
|
||||
"; {asked_for_date} fetch(es) delegated to a subagent asked for the \
|
||||
page's date, which the task did not and the skill does"
|
||||
));
|
||||
}
|
||||
Verdict::PassWith(why)
|
||||
}
|
||||
|
||||
/// `arxiv-daily` forbids searching arXiv — the harvest already ran.
|
||||
///
|
||||
/// This is the one boundary we have watched an agent cross in production, so it
|
||||
@@ -1047,6 +1215,120 @@ mod tests {
|
||||
);
|
||||
}
|
||||
|
||||
fn bash(cmd: &str) -> ToolEvidence {
|
||||
ToolEvidence {
|
||||
tool: "Bash".into(),
|
||||
path: None,
|
||||
input: json!({ "command": cmd }),
|
||||
response: serde_json::Value::Null,
|
||||
}
|
||||
}
|
||||
|
||||
fn spawn(prompt: &str) -> ToolEvidence {
|
||||
ToolEvidence {
|
||||
tool: "Agent".into(),
|
||||
path: None,
|
||||
input: json!({ "prompt": prompt, "subagent_type": "general-purpose" }),
|
||||
response: serde_json::Value::Null,
|
||||
}
|
||||
}
|
||||
|
||||
/// The shape mission 01a09b42 recorded: the parent read the skill, then
|
||||
/// sent each primary source to a subagent and asked for the date — which
|
||||
/// the task never did and the skill's "undated is a finding" does.
|
||||
#[test]
|
||||
fn triage_passes_on_primary_sources_and_reports_the_date_fingerprint() {
|
||||
let tools = vec![
|
||||
spawn(
|
||||
"Fetch the following URLs and return their full text content. Return the \
|
||||
URL, date if visible, and the key content.\n1. https://arxiv.org/abs/2309.15217 \
|
||||
(RAGAS paper)\n2. https://arxiv.org/abs/2311.09476",
|
||||
),
|
||||
bash("curl -s \"https://arxiv.org/abs/2309.01431\" | head -200"),
|
||||
];
|
||||
let ev = Evidence::new("", &tools);
|
||||
match triage_compliance(&ev) {
|
||||
Verdict::PassWith(why) => {
|
||||
assert!(why.contains("3 primary source(s)"), "{why}");
|
||||
assert!(why.contains("asked for the page's date"), "{why}");
|
||||
}
|
||||
other => panic!("expected a pass with evidence, got {other:?}"),
|
||||
}
|
||||
}
|
||||
|
||||
/// The shape the `index` runs recorded — inline curls, no delegation. The
|
||||
/// agent still read primary sources, so it still complied; the date
|
||||
/// fingerprint is extra evidence, never a requirement.
|
||||
#[test]
|
||||
fn triage_passes_on_inline_curls_without_the_date_clause() {
|
||||
let tools = vec![
|
||||
bash("curl -sL https://arxiv.org/abs/2204.04745"),
|
||||
bash("ls -la research/"),
|
||||
];
|
||||
match triage_compliance(&Evidence::new("", &tools)) {
|
||||
Verdict::PassWith(why) => assert!(!why.contains("date"), "{why}"),
|
||||
other => panic!("{other:?}"),
|
||||
}
|
||||
}
|
||||
|
||||
/// The one violation the check can see: reading a restatement of a
|
||||
/// restatement instead of the source it restates.
|
||||
#[test]
|
||||
fn triage_fails_on_an_aggregator() {
|
||||
let tools = vec![
|
||||
bash("curl -s https://arxiv.org/abs/2309.15217"),
|
||||
spawn("Fetch https://medium.com/@someone/rag-eval-explained-2024 and summarise"),
|
||||
];
|
||||
match triage_compliance(&Evidence::new("", &tools)) {
|
||||
Verdict::Fail(why) => assert!(why.contains("medium.com"), "{why}"),
|
||||
other => panic!("{other:?}"),
|
||||
}
|
||||
}
|
||||
|
||||
/// One-sided, both ways. Nothing fetched is nothing to triage; something
|
||||
/// fetched that the lists cannot rank is unranked, not a violation.
|
||||
#[test]
|
||||
fn triage_is_silent_where_it_cannot_see() {
|
||||
let none: Vec<ToolEvidence> = vec![];
|
||||
assert!(matches!(
|
||||
triage_compliance(&Evidence::new("", &none)),
|
||||
Verdict::NotObservable(_)
|
||||
));
|
||||
let no_fetch = vec![bash("cargo test"), bash("git status")];
|
||||
assert!(matches!(
|
||||
triage_compliance(&Evidence::new("", &no_fetch)),
|
||||
Verdict::NotApplicable
|
||||
));
|
||||
let unranked = vec![bash("curl -s https://docs.example-vendor.io/eval/guide")];
|
||||
assert!(matches!(
|
||||
triage_compliance(&Evidence::new("", &unranked)),
|
||||
Verdict::NotObservable(_)
|
||||
));
|
||||
}
|
||||
|
||||
/// A URL inside JSON is followed by a quote, and one at the end of a
|
||||
/// sentence by a full stop. Neither is part of the URL.
|
||||
#[test]
|
||||
fn fetched_urls_stop_at_the_right_character() {
|
||||
let tools = vec![spawn(
|
||||
"Fetch \"https://arxiv.org/abs/1\" then https://doi.org/10.1/x. Done.",
|
||||
)];
|
||||
assert_eq!(
|
||||
fetched_urls(&Evidence::new("", &tools)),
|
||||
vec!["https://arxiv.org/abs/1", "https://doi.org/10.1/x"]
|
||||
);
|
||||
}
|
||||
|
||||
/// `PassWith` must be indistinguishable from `Pass` to a reader keyed on
|
||||
/// the tag, or every consumer of the report grows a fourth branch.
|
||||
#[test]
|
||||
fn a_pass_with_evidence_serialises_under_the_pass_tag() {
|
||||
let v = serde_json::to_value(Verdict::PassWith("saw it".into())).unwrap();
|
||||
assert_eq!(v["verdict"], "pass");
|
||||
assert_eq!(v["why"], "saw it");
|
||||
assert_eq!(Verdict::PassWith("x".into()).label(), Verdict::Pass.label());
|
||||
}
|
||||
|
||||
/// The whole loop, end to end: the index writes a uri, the agent reads that
|
||||
/// exact uri back, and the scorer recovers the skill's name from it.
|
||||
///
|
||||
|
||||
Reference in New Issue
Block a user