fix(skills): every team-template skill binding now resolves
55 of 85 role skill bindings pointed at skills that were never authored,
so 10 of 11 team templates bound a smaller context bundle than their role
prompts assumed. Three roles bound nothing at all (gpu.bench_engineer,
threejs.shader_author, threejs.perf_engineer) while their prompts described
procedures they had no way to read.
The loader comment at team_template_loader.rs:167 already diagnosed this —
snake_case slugs in TOML against kebab-case skill files — and it was
half-fixed: the kebab names were corrected, the snake_case ones left.
It was invisible because both existing tests assert authored ⊆ referenced
(30/30, green) and the second explicitly declines to check the other
direction. So the failing half was the half nobody asserted.
Resolved every name by one of three explicit choices:
- 23 skills authored where the role genuinely needed the procedure
(gpu, threejs, research, analysis, frontend, mobile, backend, platform)
- renames onto authored skills where one existed in substance, including
the four-near-duplicate cases that collapse onto one real skill
- 22 aspirational references deleted — a binding an agent cannot read is
a promise, not a capability
Two tests now hold it. The unit test checks referenced ⊆ authored against
the files. The new integration test runs both loaders in boot order and
asserts the bindings survive the trip through the database, which is a
different question: resolution goes through skills_catalog rows, so a skill
file that exists but fails to ingest still leaves the role empty.
Negative controls: the unit test failed naming all 55; the integration test
fails naming the exact role when one name is reverted.
threejs.shader_author and .perf_engineer gained a second and third skill
after the collapse — pin_in_context pins idx < 2, so a role left with one
skill silently pins less than the policy intends.
Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
co-authored by
Claude Opus 5
parent
ba98c29481
commit
4358964c05
@@ -0,0 +1,54 @@
|
||||
---
|
||||
name: mobile-e2e-and-simulators
|
||||
description: Running mobile tests that mean something — simulator versus device, and what only fails on real hardware.
|
||||
when_to_use: You are the tester on a mobile team setting up or debugging end-to-end runs.
|
||||
tags: [mobile, testing]
|
||||
---
|
||||
|
||||
# The simulator is not the device
|
||||
|
||||
Simulators are excellent for layout and flow, and actively misleading for
|
||||
anything touching hardware, permissions or performance. Know which question you
|
||||
are answering.
|
||||
|
||||
## What the simulator answers honestly
|
||||
|
||||
Layout across screen sizes, navigation flows, most business logic, accessibility
|
||||
labels, localisation. Run these in CI on simulators — they are fast, parallel and
|
||||
deterministic.
|
||||
|
||||
## What only a real device answers
|
||||
|
||||
- **Performance.** A simulator uses your host CPU and GPU. Frame rate, memory
|
||||
pressure and battery there are meaningless.
|
||||
- **Permissions and their denial paths.** The interesting flow is "user said no",
|
||||
and simulator permission behaviour differs.
|
||||
- **Camera, GPS, biometrics, push.** Approximated or absent.
|
||||
- **Network transitions.** Wi-Fi to cellular, offline, captive portals. This is
|
||||
where the network layer's assumptions surface.
|
||||
- **Keyboard behaviour.** Third-party keyboards and predictive input change
|
||||
layout in ways the simulator's keyboard does not.
|
||||
|
||||
## Determinism in E2E
|
||||
|
||||
The mobile equivalents of the browser rules: select by accessibility id rather
|
||||
than by position, wait on conditions rather than durations, and reset app state
|
||||
between tests. A test depending on the previous test's leftover state is the
|
||||
most common cause of a suite that passes locally and fails in CI.
|
||||
|
||||
Cold start versus warm start matters more than on web: a test that only ever
|
||||
runs warm never exercises launch-path initialisation, which is where a
|
||||
surprising share of crashes live.
|
||||
|
||||
## Keep the CI matrix honest
|
||||
|
||||
Two simulators — the smallest supported and the newest — catch most layout
|
||||
regressions cheaply. Add one physical device for the release candidate, not for
|
||||
every commit; a device farm on every push is expensive and rarely finds what the
|
||||
simulator missed except on the axes listed above.
|
||||
|
||||
## Record which it was
|
||||
|
||||
A test result without its target is unactionable. "Passed on iPhone 15 simulator,
|
||||
iOS 18" and "passed on a physical Pixel 6" are different claims, and only the
|
||||
second one says anything about performance.
|
||||
Reference in New Issue
Block a user