docs: the delivery A/B, and the first retrieval nobody asked for
Runs 9 and 10: identical task text, one server process, and a task that never
mentions skills, MCP or retrieval. Run 8 demonstrated the instrument, but its
retrieval was instructed by the task — it showed the pipe worked, not that an
agent would judge relevance.
Under `index`, two of four skills were fetched, and attribution is the part
that matters:
Solveig (lead_researcher) -> web-search-triage
Olamide (report_writer) -> scientific-writing-conventions
Each agent reached for the skill bound to its OWN role and neither reached for
another's. An agent that fetched all four would have shown only that it could.
The regression the A/B existed to catch did not appear: 34% fewer tokens, 59
tool calls against 89, both arms passed the independent judge, and the
deliverables came out slightly larger rather than thinner.
Two readings the data does not support, recorded because the first draft of
this section made one of them:
- Every `tool.call` in a phase carries the DRAIN timestamp, not the call time.
All 59 rows of run 10 read `12:48:12`. Ordering by that column said the
report writer had fetched both skills; `agent_id` says otherwise.
- The prompt saving is 15-43%, not an order of magnitude. Skill bodies are a
minority of a turn prompt. Progressive disclosure is worth doing for Trigger,
not for context economy.
`workspace-repo-commit-protocol` scores Trigger=FAIL beside boundary=pass: it
behaved correctly without reading the rule. That verdict is left standing and
argued with in the text rather than tuned away.
n=1 per arm. A signal, not a rate.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
co-authored by
Claude Opus 5
parent
22eeaa6f15
commit
accae7fa94
@@ -286,6 +286,90 @@ skill itself names. It is recorded here rather than quietly corrected, because
|
||||
a measurement that hides its own false positives cannot be trusted about
|
||||
anyone else's.
|
||||
|
||||
## Runs 9 and 10 — the delivery A/B, and the first unprompted Trigger
|
||||
|
||||
Two `research_only` missions, **identical task text**, launched minutes apart
|
||||
against one server process. The task never mentions skills, MCP or retrieval —
|
||||
which is the whole point. Run 8 demonstrated the instrument, but its retrieval
|
||||
was *instructed by the task*; nothing there showed an agent judging relevance.
|
||||
|
||||
| | run 9 (`inline`) | run 10 (`index`) |
|
||||
|---|---|---|
|
||||
| skills delivered | 4 | 4 |
|
||||
| prompt bytes (3 turns) | 8642 / 7689 / 10165 | 7329 / 5524 / 5762 |
|
||||
| tool calls recorded | 89 | **59** |
|
||||
| tokens (3 steps) | 52,191 | **35,090** |
|
||||
| `ReadMcpResourceTool` | 0 | **2** |
|
||||
| judge verdict | met | met |
|
||||
|
||||
Per skill:
|
||||
|
||||
| skill | run 9 trigger | run 10 trigger | boundary (both) |
|
||||
|---|---|---|---|
|
||||
| `web-search-triage` | not observable | **pass** | n/a |
|
||||
| `scientific-writing-conventions` | not observable | **pass** | n/a |
|
||||
| `structured-paper-summary` | not observable | n/a | n/a |
|
||||
| `workspace-repo-commit-protocol` | not observable | **FAIL** | pass |
|
||||
|
||||
### What the retrievals actually show
|
||||
|
||||
Attribution is the load-bearing detail, and it is only available because
|
||||
`attribute_sessions` maps a phase's tool calls back to the agent that made
|
||||
them:
|
||||
|
||||
Solveig (lead_researcher) -> skill:global/web-search-triage
|
||||
Olamide (report_writer) -> skill:global/scientific-writing-conventions
|
||||
|
||||
**Each agent reached for the skill bound to its own role, and neither reached
|
||||
for another role's.** That is a relevance judgement, not a sweep — an agent
|
||||
that fetched all four would have shown nothing except that it could.
|
||||
|
||||
Two skills went unread. `structured-paper-summary` scores `NotApplicable`: its
|
||||
holder passed over it and nothing in that phase could check whether it should
|
||||
have, so calling that a miss would punish correct triage.
|
||||
`workspace-repo-commit-protocol` scores **Fail**, and that verdict is the one
|
||||
to argue with: the agent never fetched the procedure, yet every write it made
|
||||
landed inside the checkout, so `boundary` passes. It behaved correctly without
|
||||
reading the rule. Under the rule as written — an applicable skill offered and
|
||||
not read is a Trigger failure — that is a fail, and the pass beside it is the
|
||||
honest counterweight rather than a contradiction.
|
||||
|
||||
### The regression the A/B existed to catch did not appear
|
||||
|
||||
Progressive disclosure can only cost Compliance: under `inline` the procedure
|
||||
sits in front of the model whether or not it noticed it applied. It did not
|
||||
cost anything measurable here. The `index` arm used **34% fewer tokens and 30
|
||||
fewer tool calls**, both arms passed the independent judge, and the deliverables
|
||||
came out slightly larger, not thinner:
|
||||
|
||||
research/speculative-decoding.md 4870 -> 6182
|
||||
research/evidence.md 10823 -> 11727
|
||||
research/evidence-check.md 3340 -> 5336
|
||||
research/questions.md 1894 -> 2402
|
||||
research/REPORT.md 9867 -> 7901
|
||||
|
||||
The `Edit` count is where the arms differ most (31 -> 5). That is a behavioural
|
||||
difference the measurement did not predict and cannot explain from one run
|
||||
each.
|
||||
|
||||
### Two readings this data does not support
|
||||
|
||||
**The prompt saving is small.** Skill bodies are a minority of a turn prompt,
|
||||
so withholding them cut 15-43% of the bytes, not the order of magnitude the
|
||||
framing suggests. Progressive disclosure is worth doing for Trigger, not for
|
||||
context economy.
|
||||
|
||||
**Every `tool.call` in a phase carries the same `created_at`** — the drain
|
||||
timestamp, not the call time. All 59 rows in run 10 read `12:48:12`. Ordering
|
||||
tool calls by that column produces a confident, entirely fabricated narrative;
|
||||
the first draft of this section said the report writer had fetched both skills,
|
||||
because both retrievals appeared inside its turn window. Attribution by
|
||||
`agent_id` is the real answer and it says something different.
|
||||
|
||||
**n = 1 per arm.** Two missions do not establish a rate. What they establish is
|
||||
that the axis now produces a signal at all, and that the control arm's
|
||||
structural blind spot is gone rather than papered over.
|
||||
|
||||
## Honest limits
|
||||
|
||||
- **Five runs, one tier, two workflows.** Nothing here generalises to the
|
||||
|
||||
Reference in New Issue
Block a user