Files
clawmates/crates/cm-api/tests/skill_bindings.rs
T
Omar SobhandClaude Opus 5 4358964c05 fix(skills): every team-template skill binding now resolves
55 of 85 role skill bindings pointed at skills that were never authored,
so 10 of 11 team templates bound a smaller context bundle than their role
prompts assumed. Three roles bound nothing at all (gpu.bench_engineer,
threejs.shader_author, threejs.perf_engineer) while their prompts described
procedures they had no way to read.

The loader comment at team_template_loader.rs:167 already diagnosed this —
snake_case slugs in TOML against kebab-case skill files — and it was
half-fixed: the kebab names were corrected, the snake_case ones left.

It was invisible because both existing tests assert authored ⊆ referenced
(30/30, green) and the second explicitly declines to check the other
direction. So the failing half was the half nobody asserted.

Resolved every name by one of three explicit choices:

  - 23 skills authored where the role genuinely needed the procedure
    (gpu, threejs, research, analysis, frontend, mobile, backend, platform)
  - renames onto authored skills where one existed in substance, including
    the four-near-duplicate cases that collapse onto one real skill
  - 22 aspirational references deleted — a binding an agent cannot read is
    a promise, not a capability

Two tests now hold it. The unit test checks referenced ⊆ authored against
the files. The new integration test runs both loaders in boot order and
asserts the bindings survive the trip through the database, which is a
different question: resolution goes through skills_catalog rows, so a skill
file that exists but fails to ingest still leaves the role empty.

Negative controls: the unit test failed naming all 55; the integration test
fails naming the exact role when one name is reverted.

threejs.shader_author and .perf_engineer gained a second and third skill
after the collapse — pin_in_context pins idx < 2, so a role left with one
skill silently pins less than the policy intends.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-08-19 07:42:48 -07:00

101 lines
3.9 KiB
Rust

//! Every skill a team template names must survive the trip through the
//! database and come back out as a real binding.
//!
//! `team_template_loader`'s unit test checks names against the files in
//! `skills/`. That is not the same question: bindings are resolved by
//! `skills_catalog::get_by_name` against rows that `skills_loader` wrote, so a
//! skill file that exists but fails to ingest (bad frontmatter, a name that
//! does not match its filename) still leaves the role with an empty bundle —
//! the exact silent-empty failure the loader's own comment describes.
//!
//! This runs both loaders in boot order and asserts the bindings landed.
use std::collections::HashSet;
fn repo_root() -> std::path::PathBuf {
std::path::Path::new(env!("CARGO_MANIFEST_DIR"))
.join("../..")
.canonicalize()
.expect("canonicalize repo root")
}
/// Every `skills = [...]` name in every team template, deduplicated per role
/// the same way the loader binds them (slot + name).
fn referenced_bindings() -> HashSet<(String, String, String)> {
let mut out = HashSet::new();
let dir = repo_root().join("templates/teams");
for entry in std::fs::read_dir(&dir).expect("read templates/teams") {
let path = entry.expect("dir entry").path();
if path.extension().and_then(|e| e.to_str()) != Some("toml") {
continue;
}
let key = path.file_stem().unwrap().to_string_lossy().to_string();
let body = std::fs::read_to_string(&path).expect("read template");
let mut slot = String::new();
for line in body.lines() {
let t = line.trim();
if let Some(rest) = t.strip_prefix("slot") {
if let Some(v) = rest.split('"').nth(1) {
slot = v.to_string();
}
}
if t.starts_with("skills") {
if let Some(inner) = t.split_once('[').and_then(|(_, r)| r.rsplit_once(']')) {
for name in inner.0.split(',') {
let name = name.trim().trim_matches('"');
if !name.is_empty() {
out.insert((key.clone(), slot.clone(), name.to_string()));
}
}
}
}
}
}
out
}
#[tokio::test]
async fn every_template_role_skill_binds_through_the_database() {
let pool = cm_testkit::test_pool().await;
// Both loaders resolve their directory relative to the process cwd, which
// for an integration test is the crate, not the repo.
let root = repo_root();
std::env::set_var("CLAWMATES_SKILLS_DIR", root.join("skills"));
std::env::set_var("CLAWMATES_TEAM_TEMPLATES_DIR", root.join("templates/teams"));
// Boot order, from clawmates-server/src/main.rs: skills first, then the
// templates that reference them.
let n_skills = cm_api::skills_loader::load_builtins(&pool).await;
assert!(n_skills > 0, "skills_loader ingested nothing");
cm_api::team_template_loader::load_builtins(&pool).await;
let bound: HashSet<(String, String, String)> = sqlx::query_as::<_, (String, String, String)>(
"SELECT t.key, trs.slot, s.name \
FROM template_role_skills trs \
JOIN team_templates t ON t.id = trs.template_id \
JOIN skills s ON s.id = trs.skill_id",
)
.fetch_all(&pool)
.await
.expect("read template_role_skills")
.into_iter()
.collect();
let referenced = referenced_bindings();
let mut missing: Vec<String> = referenced
.difference(&bound)
.map(|(k, slot, name)| format!("{k}.{slot}{name}"))
.collect();
missing.sort();
assert!(
missing.is_empty(),
"{} template role skill binding(s) named in TOML never reached the \
database — those roles run with a smaller context bundle than their \
prompt assumes:\n {}",
missing.len(),
missing.join("\n ")
);
}