55 of 85 role skill bindings pointed at skills that were never authored,
so 10 of 11 team templates bound a smaller context bundle than their role
prompts assumed. Three roles bound nothing at all (gpu.bench_engineer,
threejs.shader_author, threejs.perf_engineer) while their prompts described
procedures they had no way to read.
The loader comment at team_template_loader.rs:167 already diagnosed this —
snake_case slugs in TOML against kebab-case skill files — and it was
half-fixed: the kebab names were corrected, the snake_case ones left.
It was invisible because both existing tests assert authored ⊆ referenced
(30/30, green) and the second explicitly declines to check the other
direction. So the failing half was the half nobody asserted.
Resolved every name by one of three explicit choices:
- 23 skills authored where the role genuinely needed the procedure
(gpu, threejs, research, analysis, frontend, mobile, backend, platform)
- renames onto authored skills where one existed in substance, including
the four-near-duplicate cases that collapse onto one real skill
- 22 aspirational references deleted — a binding an agent cannot read is
a promise, not a capability
Two tests now hold it. The unit test checks referenced ⊆ authored against
the files. The new integration test runs both loaders in boot order and
asserts the bindings survive the trip through the database, which is a
different question: resolution goes through skills_catalog rows, so a skill
file that exists but fails to ingest still leaves the role empty.
Negative controls: the unit test failed naming all 55; the integration test
fails naming the exact role when one name is reverted.
threejs.shader_author and .perf_engineer gained a second and third skill
after the collapse — pin_in_context pins idx < 2, so a role left with one
skill silently pins less than the policy intends.
Co-Authored-By: Claude Opus 5 <[email protected]>
101 lines
3.9 KiB
Rust
101 lines
3.9 KiB
Rust
//! Every skill a team template names must survive the trip through the
|
|
//! database and come back out as a real binding.
|
|
//!
|
|
//! `team_template_loader`'s unit test checks names against the files in
|
|
//! `skills/`. That is not the same question: bindings are resolved by
|
|
//! `skills_catalog::get_by_name` against rows that `skills_loader` wrote, so a
|
|
//! skill file that exists but fails to ingest (bad frontmatter, a name that
|
|
//! does not match its filename) still leaves the role with an empty bundle —
|
|
//! the exact silent-empty failure the loader's own comment describes.
|
|
//!
|
|
//! This runs both loaders in boot order and asserts the bindings landed.
|
|
|
|
use std::collections::HashSet;
|
|
|
|
fn repo_root() -> std::path::PathBuf {
|
|
std::path::Path::new(env!("CARGO_MANIFEST_DIR"))
|
|
.join("../..")
|
|
.canonicalize()
|
|
.expect("canonicalize repo root")
|
|
}
|
|
|
|
/// Every `skills = [...]` name in every team template, deduplicated per role
|
|
/// the same way the loader binds them (slot + name).
|
|
fn referenced_bindings() -> HashSet<(String, String, String)> {
|
|
let mut out = HashSet::new();
|
|
let dir = repo_root().join("templates/teams");
|
|
for entry in std::fs::read_dir(&dir).expect("read templates/teams") {
|
|
let path = entry.expect("dir entry").path();
|
|
if path.extension().and_then(|e| e.to_str()) != Some("toml") {
|
|
continue;
|
|
}
|
|
let key = path.file_stem().unwrap().to_string_lossy().to_string();
|
|
let body = std::fs::read_to_string(&path).expect("read template");
|
|
let mut slot = String::new();
|
|
for line in body.lines() {
|
|
let t = line.trim();
|
|
if let Some(rest) = t.strip_prefix("slot") {
|
|
if let Some(v) = rest.split('"').nth(1) {
|
|
slot = v.to_string();
|
|
}
|
|
}
|
|
if t.starts_with("skills") {
|
|
if let Some(inner) = t.split_once('[').and_then(|(_, r)| r.rsplit_once(']')) {
|
|
for name in inner.0.split(',') {
|
|
let name = name.trim().trim_matches('"');
|
|
if !name.is_empty() {
|
|
out.insert((key.clone(), slot.clone(), name.to_string()));
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
out
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn every_template_role_skill_binds_through_the_database() {
|
|
let pool = cm_testkit::test_pool().await;
|
|
|
|
// Both loaders resolve their directory relative to the process cwd, which
|
|
// for an integration test is the crate, not the repo.
|
|
let root = repo_root();
|
|
std::env::set_var("CLAWMATES_SKILLS_DIR", root.join("skills"));
|
|
std::env::set_var("CLAWMATES_TEAM_TEMPLATES_DIR", root.join("templates/teams"));
|
|
|
|
// Boot order, from clawmates-server/src/main.rs: skills first, then the
|
|
// templates that reference them.
|
|
let n_skills = cm_api::skills_loader::load_builtins(&pool).await;
|
|
assert!(n_skills > 0, "skills_loader ingested nothing");
|
|
cm_api::team_template_loader::load_builtins(&pool).await;
|
|
|
|
let bound: HashSet<(String, String, String)> = sqlx::query_as::<_, (String, String, String)>(
|
|
"SELECT t.key, trs.slot, s.name \
|
|
FROM template_role_skills trs \
|
|
JOIN team_templates t ON t.id = trs.template_id \
|
|
JOIN skills s ON s.id = trs.skill_id",
|
|
)
|
|
.fetch_all(&pool)
|
|
.await
|
|
.expect("read template_role_skills")
|
|
.into_iter()
|
|
.collect();
|
|
|
|
let referenced = referenced_bindings();
|
|
let mut missing: Vec<String> = referenced
|
|
.difference(&bound)
|
|
.map(|(k, slot, name)| format!("{k}.{slot} → {name}"))
|
|
.collect();
|
|
missing.sort();
|
|
|
|
assert!(
|
|
missing.is_empty(),
|
|
"{} template role skill binding(s) named in TOML never reached the \
|
|
database — those roles run with a smaller context bundle than their \
|
|
prompt assumes:\n {}",
|
|
missing.len(),
|
|
missing.join("\n ")
|
|
);
|
|
}
|