fix(skills): every team-template skill binding now resolves
55 of 85 role skill bindings pointed at skills that were never authored,
so 10 of 11 team templates bound a smaller context bundle than their role
prompts assumed. Three roles bound nothing at all (gpu.bench_engineer,
threejs.shader_author, threejs.perf_engineer) while their prompts described
procedures they had no way to read.
The loader comment at team_template_loader.rs:167 already diagnosed this —
snake_case slugs in TOML against kebab-case skill files — and it was
half-fixed: the kebab names were corrected, the snake_case ones left.
It was invisible because both existing tests assert authored ⊆ referenced
(30/30, green) and the second explicitly declines to check the other
direction. So the failing half was the half nobody asserted.
Resolved every name by one of three explicit choices:
- 23 skills authored where the role genuinely needed the procedure
(gpu, threejs, research, analysis, frontend, mobile, backend, platform)
- renames onto authored skills where one existed in substance, including
the four-near-duplicate cases that collapse onto one real skill
- 22 aspirational references deleted — a binding an agent cannot read is
a promise, not a capability
Two tests now hold it. The unit test checks referenced ⊆ authored against
the files. The new integration test runs both loaders in boot order and
asserts the bindings survive the trip through the database, which is a
different question: resolution goes through skills_catalog rows, so a skill
file that exists but fails to ingest still leaves the role empty.
Negative controls: the unit test failed naming all 55; the integration test
fails naming the exact role when one name is reverted.
threejs.shader_author and .perf_engineer gained a second and third skill
after the collapse — pin_in_context pins idx < 2, so a role left with one
skill silently pins less than the policy intends.
Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
Omar Sobh
co-authored by
Claude Opus 5
Claude Opus 5
parent
ba98c29481
commit
4358964c05
@@ -306,6 +306,34 @@ mod tests {
|
|||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// EVERY referenced name must resolve to an authored skill.
|
||||||
|
///
|
||||||
|
/// The other direction, and the one that was missing. Both existing tests
|
||||||
|
/// assert `authored ⊆ referenced` — true of all 30 authored skills, so both
|
||||||
|
/// passed while 55 of 85 bindings resolved to nothing and ten roles ran with
|
||||||
|
/// an empty context bundle.
|
||||||
|
///
|
||||||
|
/// The old comment on the test below called the gap "deliberately
|
||||||
|
/// aspirational". An aspirational binding is indistinguishable at runtime
|
||||||
|
/// from a typo: `get_by_name` returns Ok(None), the loader logs a line
|
||||||
|
/// nobody reads, and the role ships without the instructions its prompt
|
||||||
|
/// assumes it has. If a skill is worth naming it is worth authoring, and if
|
||||||
|
/// it is not, the name should not be in the template.
|
||||||
|
#[test]
|
||||||
|
fn every_referenced_skill_resolves_to_an_authored_one() {
|
||||||
|
let mut authored = HashSet::new();
|
||||||
|
authored_skill_names(&repo_root().join("skills"), &mut authored);
|
||||||
|
let referenced = referenced_skill_names();
|
||||||
|
let mut missing: Vec<_> = referenced.difference(&authored).cloned().collect();
|
||||||
|
missing.sort();
|
||||||
|
assert!(
|
||||||
|
missing.is_empty(),
|
||||||
|
"{} referenced skill(s) bind to nothing — the role gets no instructions \
|
||||||
|
and nothing errors: {missing:#?}",
|
||||||
|
missing.len()
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
/// A referenced name that matches no authored skill binds to nothing. Some
|
/// A referenced name that matches no authored skill binds to nothing. Some
|
||||||
/// are deliberately aspirational, so this asserts the *resolvable* ones
|
/// are deliberately aspirational, so this asserts the *resolvable* ones
|
||||||
/// stay resolvable rather than demanding every name exist.
|
/// stay resolvable rather than demanding every name exist.
|
||||||
|
|||||||
@@ -0,0 +1,100 @@
|
|||||||
|
//! Every skill a team template names must survive the trip through the
|
||||||
|
//! database and come back out as a real binding.
|
||||||
|
//!
|
||||||
|
//! `team_template_loader`'s unit test checks names against the files in
|
||||||
|
//! `skills/`. That is not the same question: bindings are resolved by
|
||||||
|
//! `skills_catalog::get_by_name` against rows that `skills_loader` wrote, so a
|
||||||
|
//! skill file that exists but fails to ingest (bad frontmatter, a name that
|
||||||
|
//! does not match its filename) still leaves the role with an empty bundle —
|
||||||
|
//! the exact silent-empty failure the loader's own comment describes.
|
||||||
|
//!
|
||||||
|
//! This runs both loaders in boot order and asserts the bindings landed.
|
||||||
|
|
||||||
|
use std::collections::HashSet;
|
||||||
|
|
||||||
|
fn repo_root() -> std::path::PathBuf {
|
||||||
|
std::path::Path::new(env!("CARGO_MANIFEST_DIR"))
|
||||||
|
.join("../..")
|
||||||
|
.canonicalize()
|
||||||
|
.expect("canonicalize repo root")
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Every `skills = [...]` name in every team template, deduplicated per role
|
||||||
|
/// the same way the loader binds them (slot + name).
|
||||||
|
fn referenced_bindings() -> HashSet<(String, String, String)> {
|
||||||
|
let mut out = HashSet::new();
|
||||||
|
let dir = repo_root().join("templates/teams");
|
||||||
|
for entry in std::fs::read_dir(&dir).expect("read templates/teams") {
|
||||||
|
let path = entry.expect("dir entry").path();
|
||||||
|
if path.extension().and_then(|e| e.to_str()) != Some("toml") {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
let key = path.file_stem().unwrap().to_string_lossy().to_string();
|
||||||
|
let body = std::fs::read_to_string(&path).expect("read template");
|
||||||
|
let mut slot = String::new();
|
||||||
|
for line in body.lines() {
|
||||||
|
let t = line.trim();
|
||||||
|
if let Some(rest) = t.strip_prefix("slot") {
|
||||||
|
if let Some(v) = rest.split('"').nth(1) {
|
||||||
|
slot = v.to_string();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if t.starts_with("skills") {
|
||||||
|
if let Some(inner) = t.split_once('[').and_then(|(_, r)| r.rsplit_once(']')) {
|
||||||
|
for name in inner.0.split(',') {
|
||||||
|
let name = name.trim().trim_matches('"');
|
||||||
|
if !name.is_empty() {
|
||||||
|
out.insert((key.clone(), slot.clone(), name.to_string()));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
out
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn every_template_role_skill_binds_through_the_database() {
|
||||||
|
let pool = cm_testkit::test_pool().await;
|
||||||
|
|
||||||
|
// Both loaders resolve their directory relative to the process cwd, which
|
||||||
|
// for an integration test is the crate, not the repo.
|
||||||
|
let root = repo_root();
|
||||||
|
std::env::set_var("CLAWMATES_SKILLS_DIR", root.join("skills"));
|
||||||
|
std::env::set_var("CLAWMATES_TEAM_TEMPLATES_DIR", root.join("templates/teams"));
|
||||||
|
|
||||||
|
// Boot order, from clawmates-server/src/main.rs: skills first, then the
|
||||||
|
// templates that reference them.
|
||||||
|
let n_skills = cm_api::skills_loader::load_builtins(&pool).await;
|
||||||
|
assert!(n_skills > 0, "skills_loader ingested nothing");
|
||||||
|
cm_api::team_template_loader::load_builtins(&pool).await;
|
||||||
|
|
||||||
|
let bound: HashSet<(String, String, String)> = sqlx::query_as::<_, (String, String, String)>(
|
||||||
|
"SELECT t.key, trs.slot, s.name \
|
||||||
|
FROM template_role_skills trs \
|
||||||
|
JOIN team_templates t ON t.id = trs.template_id \
|
||||||
|
JOIN skills s ON s.id = trs.skill_id",
|
||||||
|
)
|
||||||
|
.fetch_all(&pool)
|
||||||
|
.await
|
||||||
|
.expect("read template_role_skills")
|
||||||
|
.into_iter()
|
||||||
|
.collect();
|
||||||
|
|
||||||
|
let referenced = referenced_bindings();
|
||||||
|
let mut missing: Vec<String> = referenced
|
||||||
|
.difference(&bound)
|
||||||
|
.map(|(k, slot, name)| format!("{k}.{slot} → {name}"))
|
||||||
|
.collect();
|
||||||
|
missing.sort();
|
||||||
|
|
||||||
|
assert!(
|
||||||
|
missing.is_empty(),
|
||||||
|
"{} template role skill binding(s) named in TOML never reached the \
|
||||||
|
database — those roles run with a smaller context bundle than their \
|
||||||
|
prompt assumes:\n {}",
|
||||||
|
missing.len(),
|
||||||
|
missing.join("\n ")
|
||||||
|
);
|
||||||
|
}
|
||||||
@@ -0,0 +1,53 @@
|
|||||||
|
---
|
||||||
|
name: ast-grep-repo-index
|
||||||
|
description: Building a structural map of an unfamiliar repository — entry points, module boundaries, and where the real logic lives.
|
||||||
|
when_to_use: You are mapping a codebase you did not write, before changing anything in it.
|
||||||
|
tags: [analysis, codebase]
|
||||||
|
---
|
||||||
|
|
||||||
|
# Map the shape before reading the code
|
||||||
|
|
||||||
|
An unfamiliar repository is mostly scaffolding. The goal of a first pass is to
|
||||||
|
find the small fraction that matters and ignore the rest deliberately.
|
||||||
|
|
||||||
|
## Start at the edges, not the top
|
||||||
|
|
||||||
|
```bash
|
||||||
|
rg -l 'fn main\(|async fn main' --type rust # entry points
|
||||||
|
rg -n 'route\(|\.get\(|\.post\(' | head -50 # HTTP surface
|
||||||
|
ls migrations/ | tail -20 # what the data recently grew
|
||||||
|
```
|
||||||
|
|
||||||
|
Entry points, the request surface and the schema tell you what the system *does*
|
||||||
|
in about ten minutes. Reading `lib.rs` top-down tells you how it is organised,
|
||||||
|
which is a different and less useful question early on.
|
||||||
|
|
||||||
|
## Size is a signal
|
||||||
|
|
||||||
|
```bash
|
||||||
|
find . -name '*.rs' -not -path './target/*' | xargs wc -l | sort -rn | head -20
|
||||||
|
```
|
||||||
|
|
||||||
|
The largest files are either the core logic or an accumulation nobody split.
|
||||||
|
Both are worth knowing before you touch anything nearby.
|
||||||
|
|
||||||
|
## Structural search beats text search for call graphs
|
||||||
|
|
||||||
|
Text grep for a function name matches its definition, its calls, its doc
|
||||||
|
comments and any string containing it. When you need call sites specifically,
|
||||||
|
search the shape — `ast-grep --pattern 'foo($$$)'` — or at minimum anchor the
|
||||||
|
text: `rg '\bfoo\('.
|
||||||
|
|
||||||
|
## Follow one request end to end
|
||||||
|
|
||||||
|
The single most valuable early exercise: pick one endpoint and trace it from
|
||||||
|
route registration to database and back. It crosses every layer the codebase
|
||||||
|
has, in the order the codebase thinks about them, and it tells you more than any
|
||||||
|
architecture document. See `request-lifecycle-tracing`.
|
||||||
|
|
||||||
|
## Write down the map
|
||||||
|
|
||||||
|
A map that lives only in your head has to be rebuilt by the next person. Record
|
||||||
|
entry points, the three or four modules with the real logic, and — most
|
||||||
|
valuable — what you *expected* to find and did not. The gaps between a reasonable
|
||||||
|
mental model and the actual structure are where future bugs live.
|
||||||
@@ -0,0 +1,55 @@
|
|||||||
|
---
|
||||||
|
name: git-log-forensics
|
||||||
|
description: Reading a repository's history to find when behaviour changed and why, rather than guessing from the current tree.
|
||||||
|
when_to_use: You are investigating why code is the way it is, or when a behaviour was introduced.
|
||||||
|
tags: [analysis, git]
|
||||||
|
---
|
||||||
|
|
||||||
|
# The history answers questions the tree cannot
|
||||||
|
|
||||||
|
The current tree shows what is true. It does not show what was tried, what was
|
||||||
|
reverted, or which line was load-bearing enough to be touched forty times.
|
||||||
|
|
||||||
|
## The four commands that answer most questions
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git log -S'<string>' --oneline -- <path> # when did this string appear/vanish
|
||||||
|
git log -L'<start>,<end>:<file>' # every change to these lines
|
||||||
|
git log --follow -- <file> # survives renames
|
||||||
|
git bisect start <bad> <good> # find the commit that changed it
|
||||||
|
```
|
||||||
|
|
||||||
|
`-S` (the "pickaxe") is the most under-used and the most powerful: it searches
|
||||||
|
for commits where the *count* of a string changed, so it finds the commit that
|
||||||
|
introduced a call, not every commit that mentions it. `-L` gives the biography
|
||||||
|
of a specific function.
|
||||||
|
|
||||||
|
## Churn marks risk
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git log --format= --name-only | sort | uniq -c | sort -rn | head -20
|
||||||
|
```
|
||||||
|
|
||||||
|
Files at the top are either the project's core or its problem area, and the
|
||||||
|
commit messages tell you which. A file changed in 200 commits by 15 authors is
|
||||||
|
where the next bug will be, whatever the current code looks like.
|
||||||
|
|
||||||
|
## Read the message, then distrust it
|
||||||
|
|
||||||
|
A commit message states intent. The diff states what happened. When they
|
||||||
|
disagree — "small refactor" touching thirty files, "fix typo" changing a
|
||||||
|
condition — the diff is the truth, and the disagreement is itself a finding
|
||||||
|
worth recording.
|
||||||
|
|
||||||
|
## Blame points at the last toucher, not the author
|
||||||
|
|
||||||
|
`git blame` shows who last modified a line, which after a reformat, a rename or
|
||||||
|
a lint pass is whoever ran the tool. Use `-w` (ignore whitespace) and
|
||||||
|
`-C` (detect moved code) before drawing any conclusion, and prefer `log -L` when
|
||||||
|
you want the line's history rather than its current owner.
|
||||||
|
|
||||||
|
## What to report
|
||||||
|
|
||||||
|
A forensic finding is a commit hash, a date, and the reason the change was made
|
||||||
|
if the message or its PR gives one. "This check was added in `a1b2c3d` after an
|
||||||
|
incident" is actionable. "This code looks defensive" is not.
|
||||||
@@ -0,0 +1,53 @@
|
|||||||
|
---
|
||||||
|
name: request-lifecycle-tracing
|
||||||
|
description: Following one request from entry to storage and back, so a change can be reasoned about across layers.
|
||||||
|
when_to_use: You need to understand how a specific operation works, or where to make a change that crosses layers.
|
||||||
|
tags: [analysis, codebase]
|
||||||
|
---
|
||||||
|
|
||||||
|
# One request, every layer, in order
|
||||||
|
|
||||||
|
Tracing a single path end to end is the fastest way to learn a system, and the
|
||||||
|
only reliable way to know where a cross-cutting change must land.
|
||||||
|
|
||||||
|
## The trace
|
||||||
|
|
||||||
|
```
|
||||||
|
route registration which handler, which method, what middleware
|
||||||
|
extractors / auth what must be true before the handler body runs
|
||||||
|
handler validation, then the call into real logic
|
||||||
|
domain / service the actual decision
|
||||||
|
persistence the query, the transaction boundary
|
||||||
|
response what is serialised, what is deliberately omitted
|
||||||
|
side effects events, jobs, background work
|
||||||
|
```
|
||||||
|
|
||||||
|
The last line is the one most often missed and the most likely to break: an
|
||||||
|
operation that also enqueues a job, writes an event or invalidates a cache has
|
||||||
|
consequences the response does not mention.
|
||||||
|
|
||||||
|
## Follow the data, not the call stack
|
||||||
|
|
||||||
|
Call stacks show structure; data flow shows behaviour. For each step ask what
|
||||||
|
changed shape and what was dropped. A field silently discarded between the
|
||||||
|
handler and the query is a bug that no test asserting the response will catch.
|
||||||
|
|
||||||
|
## Find the transaction boundary explicitly
|
||||||
|
|
||||||
|
Where does the transaction begin and commit? Anything outside it is not atomic
|
||||||
|
with the write, and that is where "the row exists but the event never fired"
|
||||||
|
comes from. Note it during the trace — reconstructing it later, mid-incident, is
|
||||||
|
much harder.
|
||||||
|
|
||||||
|
## Note what happens on failure
|
||||||
|
|
||||||
|
Walk it a second time asking what happens when each step fails. Errors that are
|
||||||
|
logged and swallowed are where silent failures live: the operation reports
|
||||||
|
success while a step did nothing. If a step's failure produces no observable
|
||||||
|
signal, that is a finding, not a detail.
|
||||||
|
|
||||||
|
## Record the trace with file:line
|
||||||
|
|
||||||
|
The output is a short document with a line per layer and the file and line for
|
||||||
|
each. That is what makes the next change — or the next incident — cheap, and it
|
||||||
|
is checkable by whoever reads it next.
|
||||||
@@ -0,0 +1,50 @@
|
|||||||
|
---
|
||||||
|
name: openapi-contract-first
|
||||||
|
description: Designing an HTTP API as a written contract before the handler exists, and keeping the document true afterwards.
|
||||||
|
when_to_use: You are the API designer adding or changing an endpoint.
|
||||||
|
tags: [backend, api]
|
||||||
|
---
|
||||||
|
|
||||||
|
# The contract is the deliverable; the handler implements it
|
||||||
|
|
||||||
|
Writing the schema first surfaces the disagreements while they are still cheap —
|
||||||
|
what is optional, what the error shape is, what happens on conflict.
|
||||||
|
|
||||||
|
## Decide these five before writing code
|
||||||
|
|
||||||
|
1. **Resource and method.** `POST /missions` creates; `PATCH /missions/{id}`
|
||||||
|
partially updates. If you need a verb, the resource is probably wrong.
|
||||||
|
2. **The error shape, once, for the whole API.** One envelope everywhere. A
|
||||||
|
client that must handle three error shapes will handle one and log the others.
|
||||||
|
3. **Which fields are optional, and what absent means.** Absent, `null` and empty
|
||||||
|
are three different things and clients will discover the difference in
|
||||||
|
production.
|
||||||
|
4. **The status codes you actually use.** 201 with a Location for creates, 409
|
||||||
|
for conflicts, 422 for validation. Returning 200 with `{"error": ...}` makes
|
||||||
|
every client parse the body to learn what happened.
|
||||||
|
5. **Pagination, from the first endpoint.** Retrofitting it is a breaking
|
||||||
|
change — see `api-pagination-day-1`.
|
||||||
|
|
||||||
|
## The document must stay true
|
||||||
|
|
||||||
|
A stale spec is worse than none: clients trust it and are wrong. Generate it
|
||||||
|
from the types where the framework allows, and where it is hand-written, make a
|
||||||
|
test fail when handler and document disagree.
|
||||||
|
|
||||||
|
The failure mode to design against is a field added to the response and never
|
||||||
|
to the spec. That is invisible until an integrator asks why the documented shape
|
||||||
|
does not match, which is much later and much more expensive.
|
||||||
|
|
||||||
|
## Compatibility rules
|
||||||
|
|
||||||
|
Additive is safe: new optional request fields, new response fields. Everything
|
||||||
|
else is a breaking change, including narrowing an enum, making an optional field
|
||||||
|
required, and changing a field's type — even `int` to `string` for an id.
|
||||||
|
|
||||||
|
If a break is genuinely needed, version the path. Silently changing a shape and
|
||||||
|
telling clients in a changelog is how integrations fail on a Friday.
|
||||||
|
|
||||||
|
## Examples are part of the contract
|
||||||
|
|
||||||
|
One realistic request and response per endpoint answers more questions than the
|
||||||
|
schema does, and a wrong example is caught immediately by anyone who tries it.
|
||||||
@@ -0,0 +1,61 @@
|
|||||||
|
---
|
||||||
|
name: postgres-explain-analyze
|
||||||
|
description: Reading an EXPLAIN ANALYZE plan to find why a query is slow, rather than adding indexes hopefully.
|
||||||
|
when_to_use: A query is slow, or you want to confirm an index is actually used.
|
||||||
|
tags: [backend, postgres]
|
||||||
|
---
|
||||||
|
|
||||||
|
# Read the plan; do not guess at indexes
|
||||||
|
|
||||||
|
An index added without a plan is as likely to be unused as to help, and each one
|
||||||
|
costs write throughput forever.
|
||||||
|
|
||||||
|
## Always ANALYZE, and usually BUFFERS
|
||||||
|
|
||||||
|
```sql
|
||||||
|
EXPLAIN (ANALYZE, BUFFERS, FORMAT TEXT) SELECT ...;
|
||||||
|
```
|
||||||
|
|
||||||
|
`EXPLAIN` alone shows the planner's *estimate*. `ANALYZE` executes and shows
|
||||||
|
what happened, which is the only thing worth reading. `BUFFERS` shows whether
|
||||||
|
the data came from cache or disk — a "slow" query that is entirely
|
||||||
|
`shared read` is an I/O problem, not a plan problem.
|
||||||
|
|
||||||
|
Note that `ANALYZE` actually runs the statement. Wrap a mutation in a
|
||||||
|
transaction you roll back.
|
||||||
|
|
||||||
|
## Read it inside out, and look for two things
|
||||||
|
|
||||||
|
Plans nest; the innermost node runs first. Scan for:
|
||||||
|
|
||||||
|
1. **The largest `actual time`,** not the largest estimate. That is where the
|
||||||
|
time went.
|
||||||
|
2. **Estimate versus actual rows.** `rows=10` with `actual rows=48000` is the
|
||||||
|
planner being wrong, and a wrong estimate is usually the *cause* of a bad
|
||||||
|
plan — it picked a nested loop because it expected ten rows. Fix the
|
||||||
|
statistics (`ANALYZE <table>`, or raise the statistics target) before touching
|
||||||
|
the query.
|
||||||
|
|
||||||
|
## What the node types tell you
|
||||||
|
|
||||||
|
- **Seq Scan** on a large table with a selective filter → a missing index, or a
|
||||||
|
filter the index cannot serve (a function on the column, a leading wildcard).
|
||||||
|
- **Nested Loop** with a large outer side → usually the wrong-estimate problem
|
||||||
|
above; correct rows would have produced a hash join.
|
||||||
|
- **Sort** with `Sort Method: external merge Disk` → `work_mem` too small, or an
|
||||||
|
index could provide the order for free.
|
||||||
|
- **Bitmap Heap Scan** with high `Rows Removed by Filter` → the index found
|
||||||
|
candidates the table had to reject; consider a composite or partial index.
|
||||||
|
|
||||||
|
## Confirm the index is used, not just present
|
||||||
|
|
||||||
|
After adding one, re-run the plan. An index that does not appear is dead weight:
|
||||||
|
it slows every write and helps nothing. Common causes are a type mismatch, a
|
||||||
|
function on the column, or a column order that does not match the predicate —
|
||||||
|
see `postgres-index-selection`.
|
||||||
|
|
||||||
|
## Test against realistic data volume
|
||||||
|
|
||||||
|
Plans change with size. Every plan is a Seq Scan on a thousand rows, and the
|
||||||
|
planner is right to choose it. Validate on production-shaped data or the
|
||||||
|
exercise is theatre.
|
||||||
@@ -0,0 +1,51 @@
|
|||||||
|
---
|
||||||
|
name: postgres-integration-testing
|
||||||
|
description: Testing against a real Postgres rather than a mock, with isolation that survives parallel runs.
|
||||||
|
when_to_use: You are the tester covering code that issues SQL.
|
||||||
|
tags: [backend, testing, postgres]
|
||||||
|
---
|
||||||
|
|
||||||
|
# Mock the network, never the database
|
||||||
|
|
||||||
|
A mocked database agrees with whatever the code believed. Constraints, cascades,
|
||||||
|
transaction visibility, type coercion and the SQL itself are exactly the things
|
||||||
|
that break, and a mock asserts none of them.
|
||||||
|
|
||||||
|
## Isolation is the whole problem
|
||||||
|
|
||||||
|
Parallel tests sharing a database interfere in ways that look like flaky code.
|
||||||
|
Three workable strategies:
|
||||||
|
|
||||||
|
1. **A fresh migrated database per test.** Cleanest and slowest. Right when the
|
||||||
|
suite is small or the schema is the thing under test.
|
||||||
|
2. **A transaction per test, rolled back.** Fast and well isolated, but the code
|
||||||
|
under test cannot manage its own transactions — which rules it out for
|
||||||
|
anything testing commit behaviour.
|
||||||
|
3. **Unique keys per test.** Every row keyed by a per-test uuid, no cleanup.
|
||||||
|
Scales well, and the leftover rows are useful when something fails.
|
||||||
|
|
||||||
|
Pick one per suite and say which. Mixing them produces the failures each was
|
||||||
|
meant to prevent.
|
||||||
|
|
||||||
|
## Test what only a real database can tell you
|
||||||
|
|
||||||
|
- **Constraints fire.** A unique violation, a FK failure, a check constraint —
|
||||||
|
assert the error, not just the happy path.
|
||||||
|
- **Cascades do what you think.** `ON DELETE CASCADE` reaching further than
|
||||||
|
intended is a data-loss bug that only a real delete reveals.
|
||||||
|
- **The migration applies to a populated table.** A migration tested on an empty
|
||||||
|
database has not been tested. Insert rows first, then migrate.
|
||||||
|
- **Concurrent claims are atomic.** For any `FOR UPDATE SKIP LOCKED` queue,
|
||||||
|
run two workers and assert the row was claimed once.
|
||||||
|
|
||||||
|
## Assert on the database, not only the return value
|
||||||
|
|
||||||
|
A handler can return 200 while writing nothing. Read the row back and check it.
|
||||||
|
The class of bug where the code "succeeded" and the data did not change is
|
||||||
|
invisible to a test that only inspects the response.
|
||||||
|
|
||||||
|
## Keep it fast enough to run
|
||||||
|
|
||||||
|
An integration suite nobody runs protects nothing. Share one container across
|
||||||
|
the suite rather than per test, run in parallel with proper isolation, and keep
|
||||||
|
fixtures small — realistic in shape, not in volume.
|
||||||
@@ -0,0 +1,51 @@
|
|||||||
|
---
|
||||||
|
name: a11y-checklist
|
||||||
|
description: The accessibility checks that catch most real failures, and the ones automated tools cannot make for you.
|
||||||
|
when_to_use: You are designing or testing a UI component or screen.
|
||||||
|
tags: [frontend, accessibility]
|
||||||
|
---
|
||||||
|
|
||||||
|
# Automated checks find about a third
|
||||||
|
|
||||||
|
Run axe, then do the four manual checks it cannot do. A clean axe report on an
|
||||||
|
unusable interface is the normal outcome, not a rare one.
|
||||||
|
|
||||||
|
## The four manual checks
|
||||||
|
|
||||||
|
1. **Tab through it.** Every interactive element reachable, in a sensible order,
|
||||||
|
with a visible focus ring. If focus disappears into an offscreen element or a
|
||||||
|
closed menu, the page is unusable by keyboard — the single most common real
|
||||||
|
failure.
|
||||||
|
2. **Operate it without a mouse.** Open the menu, pick an item, close it with
|
||||||
|
Escape. Dialogs trap focus while open and return it to the trigger on close.
|
||||||
|
3. **Zoom to 200%.** Text reflows, nothing is clipped, nothing overlaps. This is
|
||||||
|
also the fastest way to find fixed-height containers that will break on a
|
||||||
|
phone.
|
||||||
|
4. **Read it with a screen reader once.** VoiceOver on macOS, NVDA on Windows.
|
||||||
|
Five minutes on the main flow finds unlabelled buttons and images whose alt
|
||||||
|
text reads as a filename.
|
||||||
|
|
||||||
|
## What axe does catch
|
||||||
|
|
||||||
|
Colour contrast, missing form labels, missing alt attributes, ARIA misuse,
|
||||||
|
duplicate ids, heading-level jumps. Run it in CI — these regress constantly and
|
||||||
|
are cheap to fix at the point of change.
|
||||||
|
|
||||||
|
## The rules that prevent most issues
|
||||||
|
|
||||||
|
- **A button is a `<button>`.** A clickable `<div>` needs role, tabindex and
|
||||||
|
key handlers to be equivalent, and will get at least one of them wrong.
|
||||||
|
- **Every input has a `<label>`** with `for`, not just a placeholder. A
|
||||||
|
placeholder disappears on focus, which is when it was needed.
|
||||||
|
- **Never remove the focus outline without replacing it.** `outline: none` with
|
||||||
|
no substitute makes keyboard use impossible. Style it instead.
|
||||||
|
- **Icon-only buttons need an accessible name** — `aria-label`. Otherwise the
|
||||||
|
screen reader announces "button".
|
||||||
|
- **Don't announce with colour alone.** A red border with no text says nothing
|
||||||
|
to a colourblind user or a screen reader.
|
||||||
|
|
||||||
|
## Motion
|
||||||
|
|
||||||
|
Honour `prefers-reduced-motion`. Vestibular disorders are common and large
|
||||||
|
parallax or transform animations genuinely make people ill. One media query
|
||||||
|
disables the offending animations.
|
||||||
@@ -0,0 +1,60 @@
|
|||||||
|
---
|
||||||
|
name: playwright-e2e-patterns
|
||||||
|
description: Writing end-to-end tests that fail only when the product is broken — selectors, waiting, and isolation.
|
||||||
|
when_to_use: You are the tester on a frontend team writing or fixing browser tests.
|
||||||
|
tags: [frontend, testing]
|
||||||
|
---
|
||||||
|
|
||||||
|
# A flaky E2E test is worse than no E2E test
|
||||||
|
|
||||||
|
It trains the team to re-run rather than investigate, and the day it catches a
|
||||||
|
real bug, nobody believes it. Everything below is about determinism.
|
||||||
|
|
||||||
|
## Select by what the user sees
|
||||||
|
|
||||||
|
```ts
|
||||||
|
page.getByRole("button", { name: "Save" }) // best
|
||||||
|
page.getByLabel("Email")
|
||||||
|
page.getByTestId("mission-card") // when semantics genuinely absent
|
||||||
|
page.locator(".btn-primary > span:nth-child(2)") // never
|
||||||
|
```
|
||||||
|
|
||||||
|
Role and label selectors survive restyling and break when the UI genuinely
|
||||||
|
changes meaning — which is exactly the failure you want. CSS-path selectors
|
||||||
|
break on every refactor and pass while the button is invisible.
|
||||||
|
|
||||||
|
## Never sleep
|
||||||
|
|
||||||
|
```ts
|
||||||
|
await page.waitForTimeout(2000); // flaky, always
|
||||||
|
await expect(page.getByText("Saved")).toBeVisible(); // waits for the condition
|
||||||
|
```
|
||||||
|
|
||||||
|
Playwright's assertions retry until timeout. A fixed sleep is either longer than
|
||||||
|
needed (slow suite) or shorter than needed on a loaded CI box (flake). There is
|
||||||
|
no correct constant.
|
||||||
|
|
||||||
|
For network, wait on the response rather than a duration:
|
||||||
|
```ts
|
||||||
|
const done = page.waitForResponse(r => r.url().includes("/api/missions"));
|
||||||
|
await page.getByRole("button", { name: "Create" }).click();
|
||||||
|
await done;
|
||||||
|
```
|
||||||
|
|
||||||
|
## Each test creates what it needs
|
||||||
|
|
||||||
|
Tests that share a fixture pass in isolation and fail in parallel — or worse,
|
||||||
|
pass in the order you wrote them and fail when one is skipped. Create the data
|
||||||
|
in the test, key it with a unique id, and let cleanup be optional.
|
||||||
|
|
||||||
|
## Assert the user-visible outcome
|
||||||
|
|
||||||
|
Assert the success message and the row in the table, not the POST that fired.
|
||||||
|
A test asserting the request passes while the response is silently discarded —
|
||||||
|
which is precisely the bug the test existed to catch.
|
||||||
|
|
||||||
|
## When a test fails, look at the trace
|
||||||
|
|
||||||
|
`--trace on` records DOM snapshots, network and console for every step. Reading
|
||||||
|
the trace takes a minute and answers the question; re-running takes a minute and
|
||||||
|
answers nothing.
|
||||||
@@ -0,0 +1,55 @@
|
|||||||
|
---
|
||||||
|
name: criterion-benchmarking
|
||||||
|
description: Writing benchmarks whose numbers mean something — warmup, distributions, and the changes that are noise.
|
||||||
|
when_to_use: You are asked to benchmark a change, or to show a performance claim is real.
|
||||||
|
tags: [gpu, rust, benchmarking]
|
||||||
|
---
|
||||||
|
|
||||||
|
# A benchmark is an experiment
|
||||||
|
|
||||||
|
Most performance claims fail not because the code is slow but because the
|
||||||
|
measurement cannot support the claim.
|
||||||
|
|
||||||
|
## What criterion does for you
|
||||||
|
|
||||||
|
`criterion` runs the routine many times, discards warmup, and reports a
|
||||||
|
confidence interval rather than a single number. Take that seriously: if the
|
||||||
|
intervals for before and after overlap, **you have not measured an improvement**,
|
||||||
|
whatever the point estimates say.
|
||||||
|
|
||||||
|
```rust
|
||||||
|
fn bench_search(c: &mut Criterion) {
|
||||||
|
let index = build_index(10_000); // setup OUTSIDE the timed closure
|
||||||
|
c.bench_function("search/10k", |b| {
|
||||||
|
b.iter(|| index.search(black_box(&query), 10))
|
||||||
|
});
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Two mistakes this shape avoids:
|
||||||
|
- **Setup inside `iter`** measures the setup. If the setup must be per-iteration,
|
||||||
|
use `iter_batched` so it is excluded.
|
||||||
|
- **A missing `black_box`** lets the optimiser delete the work entirely. A
|
||||||
|
benchmark that got 400× faster after a refactor usually got deleted, not
|
||||||
|
optimised.
|
||||||
|
|
||||||
|
## Report the shape, not the headline
|
||||||
|
|
||||||
|
"34.7% faster" invites a follow-up question the number cannot answer. Give the
|
||||||
|
distribution, the input size, and the machine. A change that is 30% faster at
|
||||||
|
10k elements and 5% slower at 10M is a trade-off, and only the sweep shows it.
|
||||||
|
|
||||||
|
## What is noise
|
||||||
|
|
||||||
|
On a laptop, expect 5-10% run-to-run variance from thermal state and other
|
||||||
|
processes alone. Treat anything under that as unmeasured. If a change is
|
||||||
|
genuinely small but real, prove it by increasing iterations rather than by
|
||||||
|
asserting it — and if that is too expensive, say the change was too small to
|
||||||
|
measure rather than reporting the point estimate as fact.
|
||||||
|
|
||||||
|
## Benchmarks are also regression tests
|
||||||
|
|
||||||
|
The value is in the baseline. Record it (`--save-baseline`), compare against it
|
||||||
|
(`--baseline`), and keep the numbers with the commit that produced them.
|
||||||
|
`benchmark_snapshots` exists for exactly this — a benchmark whose history is
|
||||||
|
lost measures nothing the next time.
|
||||||
@@ -0,0 +1,60 @@
|
|||||||
|
---
|
||||||
|
name: gpu-kernel-authoring
|
||||||
|
description: Writing the same kernel across CUDA, Metal and ROCm without three divergent implementations, and the FFI boundary back into Rust.
|
||||||
|
when_to_use: You are the kernel author on a GPU team, adding or changing a compute kernel.
|
||||||
|
tags: [gpu, kernels]
|
||||||
|
---
|
||||||
|
|
||||||
|
# One kernel, three dialects
|
||||||
|
|
||||||
|
The three targets differ less than they appear. Thread indexing, memory scopes
|
||||||
|
and barriers all map onto each other; what differs is spelling and launch
|
||||||
|
configuration. Write the algorithm once, then port the spelling.
|
||||||
|
|
||||||
|
| Concept | CUDA | Metal | ROCm/HIP |
|
||||||
|
|---|---|---|---|
|
||||||
|
| thread id | `threadIdx.x` | `thread_position_in_threadgroup` | `hipThreadIdx_x` |
|
||||||
|
| block id | `blockIdx.x` | `threadgroup_position_in_grid` | `hipBlockIdx_x` |
|
||||||
|
| shared mem | `__shared__` | `threadgroup` | `__shared__` |
|
||||||
|
| barrier | `__syncthreads()` | `threadgroup_barrier(mem_flags::mem_threadgroup)` | `__syncthreads()` |
|
||||||
|
| warp/wave | 32 | 32 (SIMD-group) | **64** on CDNA, 32 on RDNA |
|
||||||
|
|
||||||
|
**The wave-size difference is the one that silently produces wrong answers.**
|
||||||
|
Any kernel that assumes 32 lanes — a warp-shuffle reduction, a ballot, an
|
||||||
|
implicit intra-warp sync — is incorrect on CDNA. Query it (`warpSize`,
|
||||||
|
`[[threads_per_simdgroup]]`) rather than hardcoding, and never rely on implicit
|
||||||
|
lockstep: it was never guaranteed on CUDA either since Volta's independent
|
||||||
|
thread scheduling.
|
||||||
|
|
||||||
|
## Get it correct on one target first
|
||||||
|
|
||||||
|
Write the scalar CPU reference, then the first GPU version, and diff the outputs
|
||||||
|
before porting anywhere. A kernel ported three ways from an unverified original
|
||||||
|
gives you three wrong answers and no baseline to find them with.
|
||||||
|
|
||||||
|
Compare with a tolerance derived from the operation, not a guess: float addition
|
||||||
|
is not associative, so a parallel reduction legitimately differs from a
|
||||||
|
sequential one. `1e-6` on an accumulation over a million elements is a failing
|
||||||
|
test that is not a bug.
|
||||||
|
|
||||||
|
## The FFI boundary
|
||||||
|
|
||||||
|
Kernels are reached from Rust through `extern "C"`. Three rules that prevent the
|
||||||
|
majority of crashes at this seam:
|
||||||
|
|
||||||
|
- **Own the allocation on one side.** Device memory allocated in C and freed in
|
||||||
|
Rust's `Drop` — or the reverse — is a lifetime nobody can read. Wrap the
|
||||||
|
device pointer in a Rust type whose `Drop` calls the same allocator's free.
|
||||||
|
- **Every launch returns a status; check it.** A kernel launch failure is
|
||||||
|
asynchronous and surfaces on the *next* synchronising call, so an unchecked
|
||||||
|
launch reports its error somewhere unrelated. Check after launch AND after
|
||||||
|
synchronise.
|
||||||
|
- **`#[repr(C)]` on every struct crossing the boundary.** Rust's default layout
|
||||||
|
is unspecified and does change.
|
||||||
|
|
||||||
|
## What to write down
|
||||||
|
|
||||||
|
A kernel's launch geometry is a decision, not a constant: record why the block
|
||||||
|
size is what it is (occupancy target, shared-memory budget, register pressure)
|
||||||
|
next to it. The next person to change the shared-memory allocation needs to know
|
||||||
|
the block size was chosen against it.
|
||||||
@@ -0,0 +1,50 @@
|
|||||||
|
---
|
||||||
|
name: gpu-profiling-workflow
|
||||||
|
description: Using Nsight, rocprof and Metal frame capture to find where a kernel actually spends time, instead of guessing.
|
||||||
|
when_to_use: You are the bench engineer on a GPU team and need to explain or improve a kernel's runtime.
|
||||||
|
tags: [gpu, profiling]
|
||||||
|
---
|
||||||
|
|
||||||
|
# Measure the machine, not your model of it
|
||||||
|
|
||||||
|
GPU intuition is unusually unreliable: the bottleneck is far more often memory
|
||||||
|
movement or occupancy than arithmetic. Profile before changing anything.
|
||||||
|
|
||||||
|
## Order of questions
|
||||||
|
|
||||||
|
1. **Is the GPU busy at all?** Kernel time versus wall time. A "slow kernel"
|
||||||
|
that occupies 8% of wall time is a host-side or transfer problem, and no
|
||||||
|
amount of kernel tuning will show up.
|
||||||
|
2. **Memory or compute bound?** Achieved bandwidth against the device peak, and
|
||||||
|
achieved FLOPs against peak. See `roofline-model` — the roofline tells you
|
||||||
|
which ceiling you are under, and therefore which optimisations can possibly
|
||||||
|
help.
|
||||||
|
3. **Occupancy?** Only after 1 and 2. Occupancy is a means, not a goal: a
|
||||||
|
kernel at 40% occupancy saturating bandwidth is finished, and raising
|
||||||
|
occupancy will not make it faster.
|
||||||
|
|
||||||
|
## The tools
|
||||||
|
|
||||||
|
- **Nsight Compute** (`ncu`) — per-kernel counters. Start with
|
||||||
|
`--set full` on ONE kernel invocation, not the whole run; it serialises and
|
||||||
|
replays kernels, so a full-application profile takes minutes and changes the
|
||||||
|
timing you were trying to measure.
|
||||||
|
- **Nsight Systems** (`nsys`) — the timeline. This is where question 1 is
|
||||||
|
answered: gaps between kernels, host-device copies, stream serialisation.
|
||||||
|
Use it *first*; `ncu` optimises a kernel that `nsys` may show is irrelevant.
|
||||||
|
- **rocprof** — the ROCm equivalent. `--stats` for the summary,
|
||||||
|
`--hip-trace`/`--hsa-trace` for the timeline.
|
||||||
|
- **Metal frame capture** — Xcode's GPU capture. Per-encoder timings and the
|
||||||
|
shader profiler's per-line cost. It is a *frame* capture: for compute work,
|
||||||
|
bracket the dispatch in a capture scope explicitly or you get nothing.
|
||||||
|
|
||||||
|
## Warm up, and say what you measured
|
||||||
|
|
||||||
|
First-call timings include JIT compilation, allocator growth and page faults —
|
||||||
|
routinely 10-100× the steady state. Discard warmup iterations and report a
|
||||||
|
distribution, not a single number: a median with a spread tells the reader
|
||||||
|
whether the change is real. See `criterion-benchmarking` for the statistics.
|
||||||
|
|
||||||
|
Record the device, driver version and clock state alongside the number. GPUs
|
||||||
|
throttle; a measurement without its conditions cannot be compared to the one
|
||||||
|
you take next month.
|
||||||
@@ -0,0 +1,54 @@
|
|||||||
|
---
|
||||||
|
name: mobile-e2e-and-simulators
|
||||||
|
description: Running mobile tests that mean something — simulator versus device, and what only fails on real hardware.
|
||||||
|
when_to_use: You are the tester on a mobile team setting up or debugging end-to-end runs.
|
||||||
|
tags: [mobile, testing]
|
||||||
|
---
|
||||||
|
|
||||||
|
# The simulator is not the device
|
||||||
|
|
||||||
|
Simulators are excellent for layout and flow, and actively misleading for
|
||||||
|
anything touching hardware, permissions or performance. Know which question you
|
||||||
|
are answering.
|
||||||
|
|
||||||
|
## What the simulator answers honestly
|
||||||
|
|
||||||
|
Layout across screen sizes, navigation flows, most business logic, accessibility
|
||||||
|
labels, localisation. Run these in CI on simulators — they are fast, parallel and
|
||||||
|
deterministic.
|
||||||
|
|
||||||
|
## What only a real device answers
|
||||||
|
|
||||||
|
- **Performance.** A simulator uses your host CPU and GPU. Frame rate, memory
|
||||||
|
pressure and battery there are meaningless.
|
||||||
|
- **Permissions and their denial paths.** The interesting flow is "user said no",
|
||||||
|
and simulator permission behaviour differs.
|
||||||
|
- **Camera, GPS, biometrics, push.** Approximated or absent.
|
||||||
|
- **Network transitions.** Wi-Fi to cellular, offline, captive portals. This is
|
||||||
|
where the network layer's assumptions surface.
|
||||||
|
- **Keyboard behaviour.** Third-party keyboards and predictive input change
|
||||||
|
layout in ways the simulator's keyboard does not.
|
||||||
|
|
||||||
|
## Determinism in E2E
|
||||||
|
|
||||||
|
The mobile equivalents of the browser rules: select by accessibility id rather
|
||||||
|
than by position, wait on conditions rather than durations, and reset app state
|
||||||
|
between tests. A test depending on the previous test's leftover state is the
|
||||||
|
most common cause of a suite that passes locally and fails in CI.
|
||||||
|
|
||||||
|
Cold start versus warm start matters more than on web: a test that only ever
|
||||||
|
runs warm never exercises launch-path initialisation, which is where a
|
||||||
|
surprising share of crashes live.
|
||||||
|
|
||||||
|
## Keep the CI matrix honest
|
||||||
|
|
||||||
|
Two simulators — the smallest supported and the newest — catch most layout
|
||||||
|
regressions cheaply. Add one physical device for the release candidate, not for
|
||||||
|
every commit; a device farm on every push is expensive and rarely finds what the
|
||||||
|
simulator missed except on the axes listed above.
|
||||||
|
|
||||||
|
## Record which it was
|
||||||
|
|
||||||
|
A test result without its target is unactionable. "Passed on iPhone 15 simulator,
|
||||||
|
iOS 18" and "passed on a physical Pixel 6" are different claims, and only the
|
||||||
|
second one says anything about performance.
|
||||||
@@ -0,0 +1,51 @@
|
|||||||
|
---
|
||||||
|
name: mobile-platform-conventions
|
||||||
|
description: Where iOS and Android genuinely differ in expected behaviour, and where a shared design is fine.
|
||||||
|
when_to_use: You are designing a screen or flow that ships on both iOS and Android.
|
||||||
|
tags: [mobile, design]
|
||||||
|
---
|
||||||
|
|
||||||
|
# Share the layout, respect the platform where it is muscle memory
|
||||||
|
|
||||||
|
Most of a design ports unchanged. A small number of behaviours are so deeply
|
||||||
|
learned that violating them reads as a bug rather than a style.
|
||||||
|
|
||||||
|
## The differences that actually matter
|
||||||
|
|
||||||
|
| | iOS | Android |
|
||||||
|
|---|---|---|
|
||||||
|
| Back | swipe from left edge; back button top-left | system back gesture/button — **must** work |
|
||||||
|
| Primary nav | tab bar, bottom | bottom nav bar (or drawer) |
|
||||||
|
| Destructive confirm | action sheet from bottom | dialog, centred |
|
||||||
|
| Text input done | "Done" / return key | often the system back dismisses |
|
||||||
|
| Sharing | share sheet | share intent |
|
||||||
|
|
||||||
|
**System back is the one to get right.** On Android, back must always do
|
||||||
|
something sensible: close the sheet, pop the screen, and at the root, exit.
|
||||||
|
A screen that traps back is broken in a way users will not report — they will
|
||||||
|
just leave.
|
||||||
|
|
||||||
|
## Safe areas are not optional
|
||||||
|
|
||||||
|
Notches, dynamic islands, home indicators and rounded corners all intrude.
|
||||||
|
Anything within 16pt of an edge needs safe-area insets, not fixed padding. Test
|
||||||
|
on a device with a notch and one without.
|
||||||
|
|
||||||
|
## Touch targets
|
||||||
|
|
||||||
|
44pt minimum on iOS, 48dp on Android. This is the most frequently violated rule
|
||||||
|
in dense interfaces and the most frequently reported as "the app feels
|
||||||
|
unreliable" rather than as a size problem.
|
||||||
|
|
||||||
|
## What NOT to differentiate
|
||||||
|
|
||||||
|
Content layout, spacing scale, typography hierarchy, colour and iconography can
|
||||||
|
and should be shared. Building two visual designs doubles the work and the bugs
|
||||||
|
for a difference users do not notice, and it is the usual reason a "platform
|
||||||
|
conventions" pass runs over.
|
||||||
|
|
||||||
|
## Test on the smallest supported device
|
||||||
|
|
||||||
|
Layouts are usually designed on a large phone and break on a small one, not the
|
||||||
|
reverse. The smallest supported screen at the largest system font size is the
|
||||||
|
worst case, and it takes one simulator run to check.
|
||||||
@@ -0,0 +1,53 @@
|
|||||||
|
---
|
||||||
|
name: brain-file-reading
|
||||||
|
description: What is actually inside a claw's .brain, which sections reach a prompt, and which are stored but never read.
|
||||||
|
when_to_use: You are inspecting an agent's brain to judge whether its definition matches what it does.
|
||||||
|
tags: [platform, brain]
|
||||||
|
---
|
||||||
|
|
||||||
|
# The brain is one HDF5 file, and most of it is not in the prompt
|
||||||
|
|
||||||
|
A `.brain` is a key/value store (`cm-brain`) holding an agent's definition. The
|
||||||
|
important thing to know before drawing conclusions from one: **stored is not the
|
||||||
|
same as used**.
|
||||||
|
|
||||||
|
## The sections
|
||||||
|
|
||||||
|
| key | reaches the model? |
|
||||||
|
|---|---|
|
||||||
|
| `identity/system_prompt` | **yes** — the system prompt |
|
||||||
|
| `identity/persona` | no |
|
||||||
|
| `identity/agent_md` | no — "how I operate", stored only |
|
||||||
|
| `identity/soul_md` | only as a fallback when `system_prompt` is empty |
|
||||||
|
| `skills/<name>` | via the role's bound skills, not from here |
|
||||||
|
| `tools/<name>` | state (`gated`/`blocked`), not prose |
|
||||||
|
| `memory/<ts_nanos>` | **yes, on the chat path** — recalled by keyword |
|
||||||
|
| `runtime/clawmates` | no — opaque JSON |
|
||||||
|
| `provenance/clawmates` | no — **a wired slot with no callers** |
|
||||||
|
|
||||||
|
`persona` and `agent_md` being stored-but-unused is deliberate and easy to
|
||||||
|
misread: an agent whose `agent_md` describes careful behaviour it does not
|
||||||
|
exhibit is not disobeying — it never saw it.
|
||||||
|
|
||||||
|
## Memory is chat-only
|
||||||
|
|
||||||
|
`remember`/`recall` are driven from the chat turn path. **Mission and phase work
|
||||||
|
never touches the brain**, so an agent that did a week of mission work has an
|
||||||
|
empty memory section. Do not conclude from that it did nothing.
|
||||||
|
|
||||||
|
Recall is BM25 keyword, not semantic: a memory phrased differently from the
|
||||||
|
query will not surface. "The agent forgot" is more often "the query did not
|
||||||
|
share words with the memory".
|
||||||
|
|
||||||
|
## Read the revisions, not just the current state
|
||||||
|
|
||||||
|
The brain is versioned (`.onion` sidecar, `commit`/`revisions`/`rollback`). The
|
||||||
|
history shows what was changed and by what — seeding, a level-up consolidation,
|
||||||
|
a human edit. A brain whose only revision is the seed has never been improved,
|
||||||
|
which is a finding about the loop, not about the agent.
|
||||||
|
|
||||||
|
## What to report
|
||||||
|
|
||||||
|
Whether the definition and the behaviour agree, and which section is responsible.
|
||||||
|
"The system prompt says X, the agent did Y" is actionable. "The brain contains
|
||||||
|
X" is not, until you have said whether X reaches the model at all.
|
||||||
@@ -0,0 +1,56 @@
|
|||||||
|
---
|
||||||
|
name: level-up-proposal-shape
|
||||||
|
description: Writing an agent-improvement proposal that a human can accept or reject on the evidence, not on the prose.
|
||||||
|
when_to_use: You are proposing a change to an agent's definition, skills or model.
|
||||||
|
tags: [platform, brain]
|
||||||
|
---
|
||||||
|
|
||||||
|
# A proposal is an argument with evidence attached
|
||||||
|
|
||||||
|
Level-up proposals go to a human review gate. The gate is the safety mechanism,
|
||||||
|
so a proposal that cannot be judged quickly either gets rubber-stamped or
|
||||||
|
ignored — both failures.
|
||||||
|
|
||||||
|
## The shape
|
||||||
|
|
||||||
|
```
|
||||||
|
OBSERVATION what the agent did, with a run or event to point at
|
||||||
|
DIAGNOSIS which part of the definition caused it
|
||||||
|
CHANGE the exact edit — the new text, not a description of it
|
||||||
|
EXPECTED what will differ next time, observably
|
||||||
|
RISK what this could make worse
|
||||||
|
```
|
||||||
|
|
||||||
|
`EXPECTED` is what makes the proposal checkable later. "Better research quality"
|
||||||
|
cannot be verified; "will cite the file it changed, as `paper-to-project-relevance`
|
||||||
|
requires" can.
|
||||||
|
|
||||||
|
## Point at evidence that will still exist
|
||||||
|
|
||||||
|
`mission_events` is retained for **seven days**. A proposal citing an event
|
||||||
|
older than that points at nothing by the time anyone re-reads it. Quote the
|
||||||
|
relevant text inline rather than referencing an id alone.
|
||||||
|
|
||||||
|
## Prefer the smallest edit that could work
|
||||||
|
|
||||||
|
A rewritten system prompt is unreviewable — a human cannot tell which of forty
|
||||||
|
changed lines caused the next difference in behaviour. One change per proposal,
|
||||||
|
so the outcome attributes to something.
|
||||||
|
|
||||||
|
## Say which layer
|
||||||
|
|
||||||
|
The same symptom has different fixes at different layers, and naming the wrong
|
||||||
|
one wastes a review cycle:
|
||||||
|
|
||||||
|
- **System prompt** — the agent's standing identity and constraints.
|
||||||
|
- **Skill** — a procedure it should follow when a situation arises.
|
||||||
|
- **Model** — capability, when the agent understood the task and could not do it.
|
||||||
|
- **Nothing** — the task was ambiguous and the agent behaved reasonably.
|
||||||
|
|
||||||
|
"Nothing" is a legitimate proposal outcome and should be written when it is true.
|
||||||
|
|
||||||
|
## Reject-worthy proposals
|
||||||
|
|
||||||
|
If you cannot name the observation, or the change is "be more careful", it is
|
||||||
|
not a proposal. Vague instructions do not change behaviour, and they accumulate
|
||||||
|
in the prompt where they crowd out the specific ones that do.
|
||||||
@@ -0,0 +1,53 @@
|
|||||||
|
---
|
||||||
|
name: metrics-baseline-comparison
|
||||||
|
description: Judging whether an agent or system change actually improved anything, against a baseline that existed first.
|
||||||
|
when_to_use: You are evaluating whether a change to an agent, prompt or model made things better.
|
||||||
|
tags: [platform, evaluation]
|
||||||
|
---
|
||||||
|
|
||||||
|
# Without a baseline there is no comparison, only an anecdote
|
||||||
|
|
||||||
|
The most common failure in agent evaluation is measuring after the change and
|
||||||
|
comparing against a memory of before.
|
||||||
|
|
||||||
|
## Record the baseline before changing anything
|
||||||
|
|
||||||
|
Whatever the metric — task success, tokens per task, wall clock, human
|
||||||
|
corrections — capture it on the current system first, over enough runs to see
|
||||||
|
the spread. Agent runs are high variance: two runs of the same task on the same
|
||||||
|
prompt can differ enormously, so a single before and a single after tells you
|
||||||
|
nothing.
|
||||||
|
|
||||||
|
## Compare like with like
|
||||||
|
|
||||||
|
Hold constant everything you are not testing: the task set, the model, the
|
||||||
|
runtime tier, the repository state. `MemoryLake on MemoryArena` is the shape to
|
||||||
|
copy — same framework, same model alias, same task samples, same scoring code,
|
||||||
|
with the memory backend the intentionally changed component.
|
||||||
|
|
||||||
|
If two things changed, the result attributes to neither.
|
||||||
|
|
||||||
|
## Variance first, effect second
|
||||||
|
|
||||||
|
Run the *unchanged* system several times to learn the noise floor. An
|
||||||
|
improvement smaller than the run-to-run spread has not been demonstrated,
|
||||||
|
however good the story is. This one discipline invalidates most informal agent
|
||||||
|
comparisons, including ones made in good faith.
|
||||||
|
|
||||||
|
## Report the distribution and the n
|
||||||
|
|
||||||
|
"9 of 40" is honest and comparable. "Significantly better" is neither. Give the
|
||||||
|
count, the denominator and the spread, and say how many runs each side had.
|
||||||
|
|
||||||
|
## Beware the metric becoming the target
|
||||||
|
|
||||||
|
An agent optimised against a judge learns the judge. If the measure is a model's
|
||||||
|
verdict, keep an independent check — a different provider family, or a
|
||||||
|
deterministic assertion the agent cannot talk its way past. A rising score with
|
||||||
|
flat real-world outcomes is the signal that this has happened.
|
||||||
|
|
||||||
|
## A negative result is a result
|
||||||
|
|
||||||
|
"No measurable difference" is worth recording and prevents the change being
|
||||||
|
proposed again in three months. Most changes do not help; a process that only
|
||||||
|
reports wins is not measuring.
|
||||||
@@ -0,0 +1,47 @@
|
|||||||
|
---
|
||||||
|
name: prior-art-search
|
||||||
|
description: Establishing whether an idea is new, and finding the work that already did it before you build.
|
||||||
|
when_to_use: You are checking novelty before proposing or building something.
|
||||||
|
tags: [research, search]
|
||||||
|
---
|
||||||
|
|
||||||
|
# Assume it has been done, and go looking
|
||||||
|
|
||||||
|
The default hypothesis for any idea is that someone published it. Searching to
|
||||||
|
confirm novelty is a different activity from searching to find prior work, and
|
||||||
|
only the second one is honest.
|
||||||
|
|
||||||
|
## Search the mechanism, not your name for it
|
||||||
|
|
||||||
|
Your framing is unlikely to match the literature's. Decompose into the
|
||||||
|
mechanism and search that:
|
||||||
|
|
||||||
|
> "agent memory that remembers what it already read"
|
||||||
|
> → deduplication, seen-set, incremental corpus, novelty detection,
|
||||||
|
> continual retrieval
|
||||||
|
|
||||||
|
Three or four vocabularies, each searched separately. A single-phrase search
|
||||||
|
returning nothing is evidence about your phrasing, not about the field.
|
||||||
|
|
||||||
|
## Follow citations in both directions
|
||||||
|
|
||||||
|
- **Backwards**: the related-work section of the closest paper you find is a
|
||||||
|
curated survey someone else already did.
|
||||||
|
- **Forwards**: who cites it. This is where the field's response lives — a
|
||||||
|
strong result from two years ago that nobody cites was probably not
|
||||||
|
reproducible, and that is worth knowing before you build on it.
|
||||||
|
|
||||||
|
Two hops in each direction from one well-chosen paper covers a field faster than
|
||||||
|
any keyword sweep.
|
||||||
|
|
||||||
|
## The negative result is the deliverable
|
||||||
|
|
||||||
|
If the search finds it has been done, say so plainly and stop. That is a
|
||||||
|
successful search that saved the build. The failure mode is a search that
|
||||||
|
concludes "novel" because the searcher wanted to build it.
|
||||||
|
|
||||||
|
## Record the search, not just the conclusion
|
||||||
|
|
||||||
|
Which terms, which databases, which date range, and what was found. A novelty
|
||||||
|
claim without its search is unfalsifiable, and six months later nobody can tell
|
||||||
|
whether the field moved or the search was thin.
|
||||||
@@ -0,0 +1,54 @@
|
|||||||
|
---
|
||||||
|
name: scientific-writing-conventions
|
||||||
|
description: Structure, hedging and claim discipline for a technical write-up that will be read by people who will check it.
|
||||||
|
when_to_use: You are drafting a paper, a technical report, or any document making empirical claims.
|
||||||
|
tags: [research, writing]
|
||||||
|
---
|
||||||
|
|
||||||
|
# Write so a sceptic can check you
|
||||||
|
|
||||||
|
The reader's question is "is this true?" Every convention below exists to let
|
||||||
|
them answer it without asking you.
|
||||||
|
|
||||||
|
## Structure
|
||||||
|
|
||||||
|
```
|
||||||
|
ABSTRACT problem, what you did, headline result WITH its number
|
||||||
|
INTRO why this matters, what was missing, contributions as a list
|
||||||
|
METHOD enough that a competent reader could reimplement it
|
||||||
|
RESULTS what happened, including what did not work
|
||||||
|
DISCUSSION what it means, what it does not, threats to validity
|
||||||
|
```
|
||||||
|
|
||||||
|
Contributions belong as an explicit list. If you cannot write three sentences
|
||||||
|
each starting "we show that...", the work is not finished.
|
||||||
|
|
||||||
|
## Hedge exactly as much as the evidence hedges
|
||||||
|
|
||||||
|
Both directions are failures. "Our method improves retrieval" from one dataset
|
||||||
|
overclaims; "may potentially suggest a possible improvement" for a result with
|
||||||
|
tight confidence intervals underclaims and reads as though you do not trust your
|
||||||
|
own experiment.
|
||||||
|
|
||||||
|
Match the language to the strength:
|
||||||
|
- measured, with intervals, replicated → *shows*, *demonstrates*
|
||||||
|
- measured once, single configuration → *indicates on X*
|
||||||
|
- consistent with, not directly tested → *is consistent with*
|
||||||
|
|
||||||
|
## Every number carries its conditions
|
||||||
|
|
||||||
|
A number without its dataset, configuration and variance is not a result. Put
|
||||||
|
them in the sentence or the table, never in a paragraph three pages away.
|
||||||
|
|
||||||
|
## Threats to validity is not a formality
|
||||||
|
|
||||||
|
State the ways you could be wrong: one seed, one hardware target, a benchmark
|
||||||
|
your method was tuned on, a baseline you implemented yourself. A reviewer will
|
||||||
|
find these; a paper that names them first is far more credible than one that
|
||||||
|
lets them be discovered.
|
||||||
|
|
||||||
|
## Say what you did not do
|
||||||
|
|
||||||
|
Scope limits are part of the contribution. "We do not evaluate on multilingual
|
||||||
|
corpora" prevents a reader assuming you did and finding out later — which costs
|
||||||
|
you far more credibility than the limitation itself.
|
||||||
@@ -0,0 +1,51 @@
|
|||||||
|
---
|
||||||
|
name: structured-paper-summary
|
||||||
|
description: Reading a paper for what it demonstrates rather than what it claims, and writing the summary in a fixed shape.
|
||||||
|
when_to_use: You are summarising a research paper for a digest, a brief, or a decision.
|
||||||
|
tags: [research, reading]
|
||||||
|
---
|
||||||
|
|
||||||
|
# Summarise the evidence, not the abstract
|
||||||
|
|
||||||
|
An abstract is the authors' pitch. The summary that is worth writing states what
|
||||||
|
was actually demonstrated, on what, against what — and where those differ from
|
||||||
|
the claim, that difference is the most valuable line in your summary.
|
||||||
|
|
||||||
|
## The shape
|
||||||
|
|
||||||
|
```
|
||||||
|
CLAIM what the authors say they show, one sentence
|
||||||
|
METHOD what they actually did — the experiment, not the framing
|
||||||
|
EVIDENCE dataset, baselines, headline numbers, and what was NOT tested
|
||||||
|
STRENGTH strong / suggestive / anecdotal, with the reason
|
||||||
|
BEARING which of our problems this touches, or none
|
||||||
|
```
|
||||||
|
|
||||||
|
`BEARING: none` is a complete and useful entry. Most papers do not apply.
|
||||||
|
|
||||||
|
## Read these four things first
|
||||||
|
|
||||||
|
1. **The baselines.** A method that beats a weak baseline has shown almost
|
||||||
|
nothing. Check whether the comparison is against the current standard or
|
||||||
|
against a version of it the authors implemented.
|
||||||
|
2. **The dataset size and shape.** A result on a million short web snippets may
|
||||||
|
not survive on ten thousand long documents. Note the scale explicitly — it is
|
||||||
|
the most common reason a transferable-looking result does not transfer.
|
||||||
|
3. **The ablations.** A paper with no ablation has not shown which part of its
|
||||||
|
contribution matters. If they claim four components and ablate none, treat
|
||||||
|
the mechanism as unproven even if the number is real.
|
||||||
|
4. **What is missing.** No error bars, one seed, one hardware configuration, no
|
||||||
|
negative results — each weakens the claim, and none of them appear in the
|
||||||
|
abstract.
|
||||||
|
|
||||||
|
## Numbers with their conditions or not at all
|
||||||
|
|
||||||
|
"12% better recall" is unusable. "12% better recall at k=10 on SIFT1M against
|
||||||
|
HNSW with M=16" can be checked, transferred or dismissed. If the conditions are
|
||||||
|
not in the paper, that is itself the finding.
|
||||||
|
|
||||||
|
## Say what would change your mind
|
||||||
|
|
||||||
|
The most useful line for a reader deciding whether to act: what result would
|
||||||
|
make this wrong, and did the authors test it? A summary that a reader can act on
|
||||||
|
tells them where the risk is, not just what the number was.
|
||||||
@@ -0,0 +1,52 @@
|
|||||||
|
---
|
||||||
|
name: web-search-triage
|
||||||
|
description: Deciding fast which search results are worth reading, and recognising the ones that are restating each other.
|
||||||
|
when_to_use: You are sweeping the open web for signal on a topic rather than reading a known source.
|
||||||
|
tags: [research, search]
|
||||||
|
---
|
||||||
|
|
||||||
|
# Most results restate a smaller number of sources
|
||||||
|
|
||||||
|
The web's response to any technical development is a primary source and then
|
||||||
|
many summaries of it. Triage is mostly about finding the primary source and
|
||||||
|
recognising the rest as one item.
|
||||||
|
|
||||||
|
## Rank by distance from the primary source
|
||||||
|
|
||||||
|
```
|
||||||
|
0 the paper, the spec, the commit, the release notes
|
||||||
|
1 the author's own blog post or thread
|
||||||
|
2 a technical write-up that adds analysis or reproduction
|
||||||
|
3 a summary of (2)
|
||||||
|
4 an aggregator restating (3)
|
||||||
|
```
|
||||||
|
|
||||||
|
Read 0 and 1. Read 2 only when it adds something the primary source did not — a
|
||||||
|
reproduction, a benchmark, a counter-argument. Everything at 3 and below is the
|
||||||
|
same item and should be recorded once, if at all.
|
||||||
|
|
||||||
|
## Signals that a page is worth reading
|
||||||
|
|
||||||
|
- It contains a number, a diff, or a reproduction someone else could run.
|
||||||
|
- It disagrees with the primary source and says why.
|
||||||
|
- It is dated, and the date is recent enough to be about the current version.
|
||||||
|
|
||||||
|
## Signals to skip
|
||||||
|
|
||||||
|
- No date, or a date that is silently the crawl date.
|
||||||
|
- Contains "revolutionary", "game-changing", or a numbered list of tools.
|
||||||
|
- Restates the abstract without adding a measurement.
|
||||||
|
- The claim is entirely in the title and the body never returns to it.
|
||||||
|
|
||||||
|
## Undated is a finding
|
||||||
|
|
||||||
|
For fast-moving topics, a page without a date cannot be triaged at all — the
|
||||||
|
same sentence can be current or two years stale. Treat undated as low priority
|
||||||
|
regardless of quality, and say so, rather than reading it and being unable to
|
||||||
|
place it.
|
||||||
|
|
||||||
|
## Stop deliberately
|
||||||
|
|
||||||
|
Sweeping has no natural end. Decide the budget first — "the top three primary
|
||||||
|
sources per subtopic" — and stop there. An exhaustive sweep that never reports
|
||||||
|
is worth less than a bounded one that does.
|
||||||
@@ -0,0 +1,59 @@
|
|||||||
|
---
|
||||||
|
name: scene-graph-planning
|
||||||
|
description: Structuring a three.js scene graph so transforms, culling and disposal stay tractable as it grows.
|
||||||
|
when_to_use: You are the scene designer laying out a three.js scene before objects are built.
|
||||||
|
tags: [threejs, architecture]
|
||||||
|
---
|
||||||
|
|
||||||
|
# The graph decides what is cheap later
|
||||||
|
|
||||||
|
Scene-graph shape determines transform cost, culling effectiveness and whether
|
||||||
|
teardown is possible. All three are painful to change once content exists.
|
||||||
|
|
||||||
|
## Group by what moves together, not by what looks similar
|
||||||
|
|
||||||
|
Every `Object3D` with a dirty transform forces a matrix recomputation down its
|
||||||
|
subtree. A graph grouped by *material* or by *asset file* means moving one
|
||||||
|
object dirties unrelated branches. Grouped by motion, a static branch stays
|
||||||
|
clean for the life of the scene.
|
||||||
|
|
||||||
|
```
|
||||||
|
world
|
||||||
|
├── static ← matrixAutoUpdate = false, set once
|
||||||
|
│ ├── terrain
|
||||||
|
│ └── props
|
||||||
|
└── dynamic
|
||||||
|
├── player
|
||||||
|
└── vehicles
|
||||||
|
```
|
||||||
|
|
||||||
|
`matrixAutoUpdate = false` on the static branch removes it from per-frame
|
||||||
|
traversal entirely. This is usually the single largest CPU win in a scene with
|
||||||
|
many objects, and it costs one line.
|
||||||
|
|
||||||
|
## Frustum culling works on bounding volumes, not intent
|
||||||
|
|
||||||
|
Culling is per-`Mesh` against its bounding sphere. Two consequences:
|
||||||
|
|
||||||
|
- **A merged mesh cannot be partially culled.** Merging 500 props into one draw
|
||||||
|
call also means all 500 are drawn whenever any part is on screen. Merge by
|
||||||
|
spatial locality, not by material alone.
|
||||||
|
- **A wrong bounding volume silently misbehaves.** After deforming geometry,
|
||||||
|
call `computeBoundingSphere()`, or the object pops out of view when its
|
||||||
|
stale sphere leaves the frustum.
|
||||||
|
|
||||||
|
## Plan disposal with the graph
|
||||||
|
|
||||||
|
WebGL resources are not garbage collected. Every geometry, material and texture
|
||||||
|
needs an explicit `dispose()`. If ownership is not planned into the graph, a
|
||||||
|
scene swap leaks GPU memory until the tab dies — see `threejs-perf-and-teardown`.
|
||||||
|
|
||||||
|
The rule that makes this tractable: **one owner per resource**, recorded where
|
||||||
|
it is created. A texture shared by twenty materials is disposed once, by the
|
||||||
|
thing that loaded it, not by whichever material is torn down first.
|
||||||
|
|
||||||
|
## Depth is not free
|
||||||
|
|
||||||
|
Deep hierarchies cost traversal on every frame. Prefer a shallow graph with
|
||||||
|
explicit groups over mirroring an asset's exported nesting, which is usually an
|
||||||
|
artefact of how it was modelled rather than how it behaves.
|
||||||
@@ -0,0 +1,55 @@
|
|||||||
|
---
|
||||||
|
name: shader-authoring-glsl-wgsl
|
||||||
|
description: Writing GLSL and WGSL shaders that compile on both, and the precision and uniform traps that only appear on some hardware.
|
||||||
|
when_to_use: You are the shader author on a three.js/WebGL team, writing or porting a shader.
|
||||||
|
tags: [threejs, shaders]
|
||||||
|
---
|
||||||
|
|
||||||
|
# GLSL and WGSL are the same ideas, spelled differently
|
||||||
|
|
||||||
|
WebGL2 takes GLSL ES 3.0; WebGPU takes WGSL. Porting is mostly mechanical, and
|
||||||
|
the mechanical parts are not where the bugs are.
|
||||||
|
|
||||||
|
| | GLSL ES 3.0 | WGSL |
|
||||||
|
|---|---|---|
|
||||||
|
| entry | `void main()` | `@fragment fn fs_main(...) -> @location(0) vec4f` |
|
||||||
|
| varyings | `in`/`out` at global scope | struct fields with `@location(n)` |
|
||||||
|
| uniforms | `uniform` block | `var<uniform>` in a bind group |
|
||||||
|
| texture | `texture(sampler2D, uv)` | `textureSample(t, s, uv)` — texture and sampler are SEPARATE |
|
||||||
|
| vec | `vec3` | `vec3f` (alias of `vec3<f32>`) |
|
||||||
|
|
||||||
|
The separated texture/sampler split is the one that changes structure: WGSL
|
||||||
|
binds them independently, so a GLSL shader using four samplers becomes four
|
||||||
|
textures plus (often) one shared sampler.
|
||||||
|
|
||||||
|
## Precision is not decoration
|
||||||
|
|
||||||
|
`mediump` in a fragment shader means **at least** 10 bits of mantissa, and on
|
||||||
|
mobile GPUs it means exactly that. A computation that is fine on a desktop
|
||||||
|
where `mediump` is silently promoted to 32-bit will band, banding-clamp or
|
||||||
|
NaN on a phone.
|
||||||
|
|
||||||
|
Rules that avoid the whole class:
|
||||||
|
- World-space positions and time accumulators are `highp`. Always. A `mediump`
|
||||||
|
time uniform visibly stutters within minutes of page load.
|
||||||
|
- Normalise in `highp`, then downcast.
|
||||||
|
- Test on a real mobile device or an emulator that honours precision. Desktop
|
||||||
|
Chrome will not show you this bug.
|
||||||
|
|
||||||
|
## Uniforms are a budget
|
||||||
|
|
||||||
|
Each uniform, varying and texture unit is a hardware-limited slot, and the
|
||||||
|
limits are much lower than desktop defaults suggest (`MAX_VARYING_VECTORS` can
|
||||||
|
be 8). Pack related scalars into a `vec4` rather than declaring four floats, and
|
||||||
|
query the limits rather than assuming.
|
||||||
|
|
||||||
|
## Shader compile errors are silent by default
|
||||||
|
|
||||||
|
three.js logs a compile failure to the console and renders black. That is
|
||||||
|
indistinguishable from a material bug, a camera bug or a culling bug. When
|
||||||
|
something renders black, check the shader log FIRST — it is a two-second check
|
||||||
|
that eliminates a large fraction of the search space.
|
||||||
|
|
||||||
|
Keep a flat-colour fallback: a shader that fails to compile should show
|
||||||
|
magenta, never black. Black is a colour the scene might legitimately be;
|
||||||
|
magenta is not.
|
||||||
@@ -0,0 +1,52 @@
|
|||||||
|
---
|
||||||
|
name: webgl-frame-profiling
|
||||||
|
description: Finding what actually costs a frame — draw calls, overdraw, shader cost — using Chrome DevTools and Spector.js.
|
||||||
|
when_to_use: You are the performance engineer on a three.js/WebGL team and a scene is dropping frames.
|
||||||
|
tags: [threejs, profiling]
|
||||||
|
---
|
||||||
|
|
||||||
|
# Find the frame's real cost before changing the scene
|
||||||
|
|
||||||
|
A dropped frame has a small number of possible causes and they are
|
||||||
|
distinguishable in minutes. Guessing usually leads to optimising geometry when
|
||||||
|
the cost was overdraw, or the reverse.
|
||||||
|
|
||||||
|
## The order
|
||||||
|
|
||||||
|
1. **CPU or GPU?** Chrome DevTools Performance panel: record 5 seconds of the
|
||||||
|
stall. Long scripting bars mean the CPU is the problem (scene-graph
|
||||||
|
traversal, matrix updates, garbage). A near-idle main thread with dropped
|
||||||
|
frames means the GPU is.
|
||||||
|
2. **How many draw calls?** `renderer.info.render.calls`. Anything in the
|
||||||
|
thousands is the answer on its own — instance, merge, or batch by material.
|
||||||
|
`renderer.info` is free and should be on screen during development.
|
||||||
|
3. **Overdraw?** Spector.js captures a frame and lists every GL call in order.
|
||||||
|
Transparent objects drawn back-to-front over the whole viewport are the
|
||||||
|
usual culprit; the same scene with `transparent: false` running fast
|
||||||
|
confirms it in one test.
|
||||||
|
4. **Shader cost?** Only now. Halve the canvas resolution: if the frame time
|
||||||
|
halves, you are fragment-bound and the shader (or overdraw) is the cost. If
|
||||||
|
it does not move, you are not.
|
||||||
|
|
||||||
|
## The resolution test is the cheapest diagnostic
|
||||||
|
|
||||||
|
`renderer.setPixelRatio(1)` versus `2` changes fragment work 4× and geometry
|
||||||
|
work not at all. That one toggle separates vertex/CPU cost from fragment cost
|
||||||
|
faster than any profiler, and it needs no tooling.
|
||||||
|
|
||||||
|
## What Spector.js is for
|
||||||
|
|
||||||
|
It is a *capture*, not a sampler: one frame, every call, with state at each
|
||||||
|
step. Use it to answer "what is this frame actually doing" — unexpected state
|
||||||
|
changes, redundant binds, a texture uploaded per frame — not to measure time.
|
||||||
|
For time, use the DevTools timeline.
|
||||||
|
|
||||||
|
## Measure the steady state
|
||||||
|
|
||||||
|
The first seconds include shader compilation, texture upload and JIT warmup.
|
||||||
|
three.js compiles a material's program on first use, so a stutter the first time
|
||||||
|
an object becomes visible is a compile, not a leak. Pre-warm with
|
||||||
|
`renderer.compile(scene, camera)` and profile after.
|
||||||
|
|
||||||
|
Record the device and the pixel ratio with any number you report. A frame time
|
||||||
|
without a resolution is not a measurement.
|
||||||
@@ -10,7 +10,7 @@ version = 1
|
|||||||
[[roles]]
|
[[roles]]
|
||||||
slot = "api_designer"
|
slot = "api_designer"
|
||||||
order_idx = 0
|
order_idx = 0
|
||||||
skills = ["decompose-int-items", "openapi_schema", "small-focused-commits", "api-pagination-day-1"]
|
skills = ["decompose-int-items", "openapi-contract-first", "small-focused-commits", "api-pagination-day-1"]
|
||||||
system_prompt = """
|
system_prompt = """
|
||||||
You are the API DESIGNER of a Backend team.
|
You are the API DESIGNER of a Backend team.
|
||||||
|
|
||||||
@@ -31,7 +31,7 @@ brain_seed = """
|
|||||||
[[roles]]
|
[[roles]]
|
||||||
slot = "db_engineer"
|
slot = "db_engineer"
|
||||||
order_idx = 1
|
order_idx = 1
|
||||||
skills = ["postgres-migrations-forward-only", "postgres-index-selection", "explain_analyze", "write-rust-current-edition", "workspace-repo-commit-protocol"]
|
skills = ["postgres-migrations-forward-only", "postgres-index-selection", "postgres-explain-analyze", "write-rust-current-edition", "workspace-repo-commit-protocol"]
|
||||||
system_prompt = """
|
system_prompt = """
|
||||||
You are the DB ENGINEER of a Backend team.
|
You are the DB ENGINEER of a Backend team.
|
||||||
|
|
||||||
@@ -55,7 +55,7 @@ brain_seed = """
|
|||||||
[[roles]]
|
[[roles]]
|
||||||
slot = "coder"
|
slot = "coder"
|
||||||
order_idx = 2
|
order_idx = 2
|
||||||
skills = ["write-rust-current-edition", "cargo_build", "cargo-test-driven-development", "workspace-repo-commit-protocol", "int-xx-marker-protocol", "rust-error-handling", "rust-async-tokio-idioms"]
|
skills = ["write-rust-current-edition", "cargo-test-driven-development", "workspace-repo-commit-protocol", "int-xx-marker-protocol", "rust-error-handling", "rust-async-tokio-idioms"]
|
||||||
system_prompt = """
|
system_prompt = """
|
||||||
You are the CODER of a Backend team.
|
You are the CODER of a Backend team.
|
||||||
|
|
||||||
@@ -68,7 +68,7 @@ brain_seed = ""
|
|||||||
[[roles]]
|
[[roles]]
|
||||||
slot = "tester"
|
slot = "tester"
|
||||||
order_idx = 3
|
order_idx = 3
|
||||||
skills = ["cargo-test-driven-development", "integration_tests_pg", "coverage_report", "tdd-red-green-refactor"]
|
skills = ["cargo-test-driven-development", "postgres-integration-testing", "tdd-red-green-refactor"]
|
||||||
system_prompt = """
|
system_prompt = """
|
||||||
You are the TESTER of a Backend team.
|
You are the TESTER of a Backend team.
|
||||||
|
|
||||||
|
|||||||
@@ -51,7 +51,7 @@ brain_seed = """
|
|||||||
[[roles]]
|
[[roles]]
|
||||||
slot = "architecture_mapper"
|
slot = "architecture_mapper"
|
||||||
order_idx = 1
|
order_idx = 1
|
||||||
skills = ["ast-grep-repo-index", "dependency-graph", "workspace-repo-commit-protocol"]
|
skills = ["ast-grep-repo-index", "workspace-repo-commit-protocol"]
|
||||||
system_prompt = """
|
system_prompt = """
|
||||||
You are the ARCHITECTURE MAPPER of a Codebase Research team.
|
You are the ARCHITECTURE MAPPER of a Codebase Research team.
|
||||||
|
|
||||||
|
|||||||
@@ -11,7 +11,7 @@ version = 1
|
|||||||
[[roles]]
|
[[roles]]
|
||||||
slot = "brain_inspector"
|
slot = "brain_inspector"
|
||||||
order_idx = 0
|
order_idx = 0
|
||||||
skills = ["brain-file-reading", "role-purpose-audit", "workspace-repo-commit-protocol"]
|
skills = ["brain-file-reading", "workspace-repo-commit-protocol"]
|
||||||
system_prompt = """
|
system_prompt = """
|
||||||
You are the BRAIN INSPECTOR of a Continuous Improvement team.
|
You are the BRAIN INSPECTOR of a Continuous Improvement team.
|
||||||
|
|
||||||
@@ -47,7 +47,7 @@ brain_seed = """
|
|||||||
[[roles]]
|
[[roles]]
|
||||||
slot = "improvement_proposer"
|
slot = "improvement_proposer"
|
||||||
order_idx = 1
|
order_idx = 1
|
||||||
skills = ["level-up-proposal-shape", "brain-consolidation", "workspace-repo-commit-protocol"]
|
skills = ["level-up-proposal-shape", "workspace-repo-commit-protocol"]
|
||||||
system_prompt = """
|
system_prompt = """
|
||||||
You are the IMPROVEMENT PROPOSER of a Continuous Improvement team.
|
You are the IMPROVEMENT PROPOSER of a Continuous Improvement team.
|
||||||
|
|
||||||
|
|||||||
@@ -10,7 +10,7 @@ version = 1
|
|||||||
[[roles]]
|
[[roles]]
|
||||||
slot = "designer"
|
slot = "designer"
|
||||||
order_idx = 0
|
order_idx = 0
|
||||||
skills = ["decompose-int-items", "design_system_check", "a11y_checklist"]
|
skills = ["decompose-int-items", "a11y-checklist"]
|
||||||
system_prompt = """
|
system_prompt = """
|
||||||
You are the DESIGNER of a Frontend team.
|
You are the DESIGNER of a Frontend team.
|
||||||
|
|
||||||
@@ -34,7 +34,7 @@ brain_seed = """
|
|||||||
[[roles]]
|
[[roles]]
|
||||||
slot = "coder"
|
slot = "coder"
|
||||||
order_idx = 1
|
order_idx = 1
|
||||||
skills = ["write_typescript_react", "tailwind-v4-idioms", "workspace-repo-commit-protocol", "int-xx-marker-protocol", "component-4-state-model", "react-19-server-components"]
|
skills = ["tailwind-v4-idioms", "workspace-repo-commit-protocol", "int-xx-marker-protocol", "component-4-state-model", "react-19-server-components"]
|
||||||
system_prompt = """
|
system_prompt = """
|
||||||
You are the CODER of a Frontend team.
|
You are the CODER of a Frontend team.
|
||||||
|
|
||||||
@@ -53,7 +53,7 @@ brain_seed = """
|
|||||||
[[roles]]
|
[[roles]]
|
||||||
slot = "tester"
|
slot = "tester"
|
||||||
order_idx = 2
|
order_idx = 2
|
||||||
skills = ["playwright_e2e", "vitest_unit", "a11y_axe", "tdd-red-green-refactor"]
|
skills = ["playwright-e2e-patterns", "a11y-checklist", "tdd-red-green-refactor"]
|
||||||
system_prompt = """
|
system_prompt = """
|
||||||
You are the TESTER of a Frontend team.
|
You are the TESTER of a Frontend team.
|
||||||
|
|
||||||
|
|||||||
@@ -32,7 +32,7 @@ brain_seed = """
|
|||||||
[[roles]]
|
[[roles]]
|
||||||
slot = "kernel_author"
|
slot = "kernel_author"
|
||||||
order_idx = 1
|
order_idx = 1
|
||||||
skills = ["write_cuda", "write_metal", "write_rocm", "write_rust_ffi", "workspace-repo-commit-protocol"]
|
skills = ["gpu-kernel-authoring", "workspace-repo-commit-protocol"]
|
||||||
system_prompt = """
|
system_prompt = """
|
||||||
You are the KERNEL AUTHOR of a GPU team.
|
You are the KERNEL AUTHOR of a GPU team.
|
||||||
|
|
||||||
@@ -52,7 +52,7 @@ brain_seed = """
|
|||||||
[[roles]]
|
[[roles]]
|
||||||
slot = "bench_engineer"
|
slot = "bench_engineer"
|
||||||
order_idx = 2
|
order_idx = 2
|
||||||
skills = ["nsight_profile", "metal_frame_capture", "rocprof", "criterion_bench"]
|
skills = ["gpu-profiling-workflow", "criterion-benchmarking"]
|
||||||
system_prompt = """
|
system_prompt = """
|
||||||
You are the BENCH ENGINEER of a GPU team.
|
You are the BENCH ENGINEER of a GPU team.
|
||||||
|
|
||||||
@@ -65,7 +65,7 @@ brain_seed = ""
|
|||||||
[[roles]]
|
[[roles]]
|
||||||
slot = "coder"
|
slot = "coder"
|
||||||
order_idx = 3
|
order_idx = 3
|
||||||
skills = ["write-rust-current-edition", "cargo_build", "cargo-test-driven-development", "workspace-repo-commit-protocol", "int-xx-marker-protocol"]
|
skills = ["write-rust-current-edition", "cargo-test-driven-development", "workspace-repo-commit-protocol", "int-xx-marker-protocol"]
|
||||||
system_prompt = """
|
system_prompt = """
|
||||||
You are the RUST-SIDE CODER. Integrate the kernel + FFI into the Rust
|
You are the RUST-SIDE CODER. Integrate the kernel + FFI into the Rust
|
||||||
library, add safe wrappers, and expose ergonomic APIs. Own the
|
library, add safe wrappers, and expose ergonomic APIs. Own the
|
||||||
|
|||||||
@@ -11,7 +11,7 @@ version = 1
|
|||||||
[[roles]]
|
[[roles]]
|
||||||
slot = "implementation_tracker"
|
slot = "implementation_tracker"
|
||||||
order_idx = 0
|
order_idx = 0
|
||||||
skills = ["git-log-forensics", "paper-citation-parsing", "workspace-repo-commit-protocol"]
|
skills = ["git-log-forensics", "workspace-repo-commit-protocol"]
|
||||||
system_prompt = """
|
system_prompt = """
|
||||||
You are the IMPLEMENTATION TRACKER of an Insight Research team.
|
You are the IMPLEMENTATION TRACKER of an Insight Research team.
|
||||||
|
|
||||||
@@ -85,7 +85,7 @@ brain_seed = """
|
|||||||
[[roles]]
|
[[roles]]
|
||||||
slot = "publication_drafter"
|
slot = "publication_drafter"
|
||||||
order_idx = 2
|
order_idx = 2
|
||||||
skills = ["scientific-writing-conventions", "figure-planning", "workspace-repo-commit-protocol", "small-focused-commits"]
|
skills = ["scientific-writing-conventions", "workspace-repo-commit-protocol", "small-focused-commits"]
|
||||||
system_prompt = """
|
system_prompt = """
|
||||||
You are the PUBLICATION DRAFTER of an Insight Research team.
|
You are the PUBLICATION DRAFTER of an Insight Research team.
|
||||||
|
|
||||||
|
|||||||
@@ -10,7 +10,7 @@ version = 1
|
|||||||
[[roles]]
|
[[roles]]
|
||||||
slot = "designer"
|
slot = "designer"
|
||||||
order_idx = 0
|
order_idx = 0
|
||||||
skills = ["decompose-int-items", "ios_hig_check", "material_you_check"]
|
skills = ["decompose-int-items", "mobile-platform-conventions"]
|
||||||
system_prompt = """
|
system_prompt = """
|
||||||
You are the DESIGNER of a Mobile team.
|
You are the DESIGNER of a Mobile team.
|
||||||
|
|
||||||
@@ -28,7 +28,7 @@ brain_seed = """
|
|||||||
[[roles]]
|
[[roles]]
|
||||||
slot = "coder"
|
slot = "coder"
|
||||||
order_idx = 1
|
order_idx = 1
|
||||||
skills = ["write_typescript_react_native", "expo-managed-vs-bare", "workspace-repo-commit-protocol", "int-xx-marker-protocol", "component-4-state-model", "rn-flashlist-perf"]
|
skills = ["expo-managed-vs-bare", "workspace-repo-commit-protocol", "int-xx-marker-protocol", "component-4-state-model", "rn-flashlist-perf"]
|
||||||
system_prompt = """
|
system_prompt = """
|
||||||
You are the CODER of a Mobile team.
|
You are the CODER of a Mobile team.
|
||||||
|
|
||||||
@@ -46,7 +46,7 @@ brain_seed = """
|
|||||||
[[roles]]
|
[[roles]]
|
||||||
slot = "tester"
|
slot = "tester"
|
||||||
order_idx = 2
|
order_idx = 2
|
||||||
skills = ["detox_e2e", "jest_unit", "ios_simulator_check", "android_emulator_check", "tdd-red-green-refactor"]
|
skills = ["mobile-e2e-and-simulators", "tdd-red-green-refactor"]
|
||||||
system_prompt = """
|
system_prompt = """
|
||||||
You are the TESTER of a Mobile team.
|
You are the TESTER of a Mobile team.
|
||||||
|
|
||||||
|
|||||||
@@ -11,7 +11,7 @@ version = 1
|
|||||||
[[roles]]
|
[[roles]]
|
||||||
slot = "domain_scout"
|
slot = "domain_scout"
|
||||||
order_idx = 0
|
order_idx = 0
|
||||||
skills = ["arxiv-query", "semantic-scholar-query", "web-search-triage", "decompose-int-items"]
|
skills = ["arxiv-daily", "web-search-triage", "decompose-int-items"]
|
||||||
system_prompt = """
|
system_prompt = """
|
||||||
You are the DOMAIN SCOUT of a Papers & Online Research team.
|
You are the DOMAIN SCOUT of a Papers & Online Research team.
|
||||||
|
|
||||||
@@ -47,7 +47,7 @@ brain_seed = """
|
|||||||
[[roles]]
|
[[roles]]
|
||||||
slot = "paper_reader"
|
slot = "paper_reader"
|
||||||
order_idx = 1
|
order_idx = 1
|
||||||
skills = ["structured-paper-summary", "pdf-text-extraction", "workspace-repo-commit-protocol"]
|
skills = ["structured-paper-summary", "workspace-repo-commit-protocol"]
|
||||||
system_prompt = """
|
system_prompt = """
|
||||||
You are the PAPER READER of a Papers & Online Research team.
|
You are the PAPER READER of a Papers & Online Research team.
|
||||||
|
|
||||||
|
|||||||
@@ -10,7 +10,7 @@ version = 1
|
|||||||
[[roles]]
|
[[roles]]
|
||||||
slot = "planner"
|
slot = "planner"
|
||||||
order_idx = 0
|
order_idx = 0
|
||||||
skills = ["read_roadmap", "decompose-int-items", "estimate_effort", "small-focused-commits"]
|
skills = ["decompose-int-items", "small-focused-commits"]
|
||||||
system_prompt = """
|
system_prompt = """
|
||||||
You are the PLANNER of a Rust SDLC team.
|
You are the PLANNER of a Rust SDLC team.
|
||||||
|
|
||||||
@@ -43,7 +43,7 @@ brain_seed = """
|
|||||||
[[roles]]
|
[[roles]]
|
||||||
slot = "coder"
|
slot = "coder"
|
||||||
order_idx = 1
|
order_idx = 1
|
||||||
skills = ["write-rust-current-edition", "cargo_build", "cargo-test-driven-development", "workspace-repo-commit-protocol", "small-focused-commits", "int-xx-marker-protocol", "rust-error-handling", "rust-async-tokio-idioms", "react-19-server-components"]
|
skills = ["write-rust-current-edition", "cargo-test-driven-development", "workspace-repo-commit-protocol", "small-focused-commits", "int-xx-marker-protocol", "rust-error-handling", "rust-async-tokio-idioms", "react-19-server-components"]
|
||||||
system_prompt = """
|
system_prompt = """
|
||||||
You are the CODER of a Rust SDLC team.
|
You are the CODER of a Rust SDLC team.
|
||||||
|
|
||||||
@@ -78,7 +78,7 @@ brain_seed = """
|
|||||||
[[roles]]
|
[[roles]]
|
||||||
slot = "tester"
|
slot = "tester"
|
||||||
order_idx = 2
|
order_idx = 2
|
||||||
skills = ["cargo-test-driven-development", "cargo_nextest", "coverage_report", "criterion_bench", "tdd-red-green-refactor"]
|
skills = ["cargo-test-driven-development", "criterion-benchmarking", "tdd-red-green-refactor"]
|
||||||
system_prompt = """
|
system_prompt = """
|
||||||
You are the TESTER of a Rust SDLC team.
|
You are the TESTER of a Rust SDLC team.
|
||||||
|
|
||||||
@@ -96,7 +96,7 @@ brain_seed = ""
|
|||||||
[[roles]]
|
[[roles]]
|
||||||
slot = "reviewer"
|
slot = "reviewer"
|
||||||
order_idx = 3
|
order_idx = 3
|
||||||
skills = ["code-review-checklist", "read_diff", "small-focused-commits", "cargo-audit-workflow", "secret-scanning-gitleaks"]
|
skills = ["code-review-checklist", "small-focused-commits", "cargo-audit-workflow", "secret-scanning-gitleaks"]
|
||||||
system_prompt = """
|
system_prompt = """
|
||||||
You are the REVIEWER of a Rust SDLC team.
|
You are the REVIEWER of a Rust SDLC team.
|
||||||
|
|
||||||
|
|||||||
@@ -10,7 +10,7 @@ version = 1
|
|||||||
[[roles]]
|
[[roles]]
|
||||||
slot = "scene_designer"
|
slot = "scene_designer"
|
||||||
order_idx = 0
|
order_idx = 0
|
||||||
skills = ["decompose-int-items", "scene_graph_planning"]
|
skills = ["decompose-int-items", "scene-graph-planning"]
|
||||||
system_prompt = """
|
system_prompt = """
|
||||||
You are the SCENE DESIGNER of a three.js team.
|
You are the SCENE DESIGNER of a three.js team.
|
||||||
|
|
||||||
@@ -28,7 +28,7 @@ brain_seed = """
|
|||||||
[[roles]]
|
[[roles]]
|
||||||
slot = "coder"
|
slot = "coder"
|
||||||
order_idx = 1
|
order_idx = 1
|
||||||
skills = ["write_typescript", "threejs-perf-and-teardown", "workspace-repo-commit-protocol", "int-xx-marker-protocol"]
|
skills = ["threejs-perf-and-teardown", "workspace-repo-commit-protocol", "int-xx-marker-protocol"]
|
||||||
system_prompt = """
|
system_prompt = """
|
||||||
You are the CODER of a three.js team.
|
You are the CODER of a three.js team.
|
||||||
|
|
||||||
@@ -47,7 +47,7 @@ brain_seed = """
|
|||||||
[[roles]]
|
[[roles]]
|
||||||
slot = "shader_author"
|
slot = "shader_author"
|
||||||
order_idx = 2
|
order_idx = 2
|
||||||
skills = ["write_glsl", "write_wgsl", "write_typescript"]
|
skills = ["shader-authoring-glsl-wgsl", "webgl-frame-profiling", "workspace-repo-commit-protocol"]
|
||||||
system_prompt = """
|
system_prompt = """
|
||||||
You are the SHADER AUTHOR. Author vertex/fragment shaders (GLSL for
|
You are the SHADER AUTHOR. Author vertex/fragment shaders (GLSL for
|
||||||
WebGL, WGSL for WebGPU). Comment mathematical steps. Provide a
|
WebGL, WGSL for WebGPU). Comment mathematical steps. Provide a
|
||||||
@@ -58,7 +58,7 @@ brain_seed = ""
|
|||||||
[[roles]]
|
[[roles]]
|
||||||
slot = "perf_engineer"
|
slot = "perf_engineer"
|
||||||
order_idx = 3
|
order_idx = 3
|
||||||
skills = ["chrome_devtools_perf", "spector_js_capture", "webgl_frame_capture"]
|
skills = ["webgl-frame-profiling", "threejs-perf-and-teardown", "shader-authoring-glsl-wgsl"]
|
||||||
system_prompt = """
|
system_prompt = """
|
||||||
You are the PERF ENGINEER. Profile with SpectorJS or Chrome DevTools
|
You are the PERF ENGINEER. Profile with SpectorJS or Chrome DevTools
|
||||||
Performance panel. Report per-frame breakdown (JS / GPU / paint) and
|
Performance panel. Report per-frame breakdown (JS / GPU / paint) and
|
||||||
|
|||||||
Reference in New Issue
Block a user