docs: the delivery A/B, and the first retrieval nobody asked for
deploy / test (push) Successful in 5m22s
deploy / build (push) Successful in 5m49s

Runs 9 and 10: identical task text, one server process, and a task that never
mentions skills, MCP or retrieval. Run 8 demonstrated the instrument, but its
retrieval was instructed by the task — it showed the pipe worked, not that an
agent would judge relevance.

Under `index`, two of four skills were fetched, and attribution is the part
that matters:

    Solveig (lead_researcher)  ->  web-search-triage
    Olamide (report_writer)    ->  scientific-writing-conventions

Each agent reached for the skill bound to its OWN role and neither reached for
another's. An agent that fetched all four would have shown only that it could.

The regression the A/B existed to catch did not appear: 34% fewer tokens, 59
tool calls against 89, both arms passed the independent judge, and the
deliverables came out slightly larger rather than thinner.

Two readings the data does not support, recorded because the first draft of
this section made one of them:

- Every `tool.call` in a phase carries the DRAIN timestamp, not the call time.
  All 59 rows of run 10 read `12:48:12`. Ordering by that column said the
  report writer had fetched both skills; `agent_id` says otherwise.
- The prompt saving is 15-43%, not an order of magnitude. Skill bodies are a
  minority of a turn prompt. Progressive disclosure is worth doing for Trigger,
  not for context economy.

`workspace-repo-commit-protocol` scores Trigger=FAIL beside boundary=pass: it
behaved correctly without reading the rule. That verdict is left standing and
argued with in the text rather than tuned away.

n=1 per arm. A signal, not a rate.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
Omar Sobh
2026-08-25 07:50:17 -05:00
co-authored by Claude Opus 5
parent 22eeaa6f15
commit accae7fa94
2 changed files with 97 additions and 7 deletions
+84
View File
@@ -286,6 +286,90 @@ skill itself names. It is recorded here rather than quietly corrected, because
a measurement that hides its own false positives cannot be trusted about
anyone else's.
## Runs 9 and 10 — the delivery A/B, and the first unprompted Trigger
Two `research_only` missions, **identical task text**, launched minutes apart
against one server process. The task never mentions skills, MCP or retrieval —
which is the whole point. Run 8 demonstrated the instrument, but its retrieval
was *instructed by the task*; nothing there showed an agent judging relevance.
| | run 9 (`inline`) | run 10 (`index`) |
|---|---|---|
| skills delivered | 4 | 4 |
| prompt bytes (3 turns) | 8642 / 7689 / 10165 | 7329 / 5524 / 5762 |
| tool calls recorded | 89 | **59** |
| tokens (3 steps) | 52,191 | **35,090** |
| `ReadMcpResourceTool` | 0 | **2** |
| judge verdict | met | met |
Per skill:
| skill | run 9 trigger | run 10 trigger | boundary (both) |
|---|---|---|---|
| `web-search-triage` | not observable | **pass** | n/a |
| `scientific-writing-conventions` | not observable | **pass** | n/a |
| `structured-paper-summary` | not observable | n/a | n/a |
| `workspace-repo-commit-protocol` | not observable | **FAIL** | pass |
### What the retrievals actually show
Attribution is the load-bearing detail, and it is only available because
`attribute_sessions` maps a phase's tool calls back to the agent that made
them:
Solveig (lead_researcher) -> skill:global/web-search-triage
Olamide (report_writer) -> skill:global/scientific-writing-conventions
**Each agent reached for the skill bound to its own role, and neither reached
for another role's.** That is a relevance judgement, not a sweep — an agent
that fetched all four would have shown nothing except that it could.
Two skills went unread. `structured-paper-summary` scores `NotApplicable`: its
holder passed over it and nothing in that phase could check whether it should
have, so calling that a miss would punish correct triage.
`workspace-repo-commit-protocol` scores **Fail**, and that verdict is the one
to argue with: the agent never fetched the procedure, yet every write it made
landed inside the checkout, so `boundary` passes. It behaved correctly without
reading the rule. Under the rule as written — an applicable skill offered and
not read is a Trigger failure — that is a fail, and the pass beside it is the
honest counterweight rather than a contradiction.
### The regression the A/B existed to catch did not appear
Progressive disclosure can only cost Compliance: under `inline` the procedure
sits in front of the model whether or not it noticed it applied. It did not
cost anything measurable here. The `index` arm used **34% fewer tokens and 30
fewer tool calls**, both arms passed the independent judge, and the deliverables
came out slightly larger, not thinner:
research/speculative-decoding.md 4870 -> 6182
research/evidence.md 10823 -> 11727
research/evidence-check.md 3340 -> 5336
research/questions.md 1894 -> 2402
research/REPORT.md 9867 -> 7901
The `Edit` count is where the arms differ most (31 -> 5). That is a behavioural
difference the measurement did not predict and cannot explain from one run
each.
### Two readings this data does not support
**The prompt saving is small.** Skill bodies are a minority of a turn prompt,
so withholding them cut 15-43% of the bytes, not the order of magnitude the
framing suggests. Progressive disclosure is worth doing for Trigger, not for
context economy.
**Every `tool.call` in a phase carries the same `created_at`** — the drain
timestamp, not the call time. All 59 rows in run 10 read `12:48:12`. Ordering
tool calls by that column produces a confident, entirely fabricated narrative;
the first draft of this section said the report writer had fetched both skills,
because both retrievals appeared inside its turn window. Attribution by
`agent_id` is the real answer and it says something different.
**n = 1 per arm.** Two missions do not establish a rate. What they establish is
that the axis now produces a signal at all, and that the control arm's
structural blind spot is gone rather than papered over.
## Honest limits
- **Five runs, one tier, two workflows.** Nothing here generalises to the