paper §7.4: complete 8-cell topology×model grid (Gemini folded in)
ci / gates (push) Has been cancelled
ci / rust (push) Has been cancelled
ci / sandbox-k8s (push) Has been cancelled
ci / frontend (push) Has been cancelled
ci / e2e (push) Has been cancelled

Gemini's project-wide key-restriction 403 is cleared (key restricted to the
Generative Language API), so Gemini rejoins the grid. Full 2×4 ranking
(pipeline/debate × Claude/GLM-4.7/Gemini/heterogeneous), judge-ranked 1-8:
D2 > P2 > P3 > P1 > D3 > PG > DG > D1.

Adds a third finding: model strength is task-dependent — Gemini-2.5-flash
placed 2nd on the §7.3 concision brief but 6th-7th here on reasoning
("generic list", "idea repeated"): concise-but-shallow. Reinforces that the
optimal config is a joint choice over {topology × per-role model × task}.
Gemini cells ran via the durable async run path (enqueue → worker → poll).

Co-Authored-By: Claude Opus 4.8 <[email protected]>
This commit is contained in:
Omar Sobh
2026-06-18 08:23:52 -07:00
co-authored by Claude Opus 4.8
parent e8486ee57c
commit 2fcec622fb
+28 -24
View File
@@ -232,32 +232,36 @@ measured rather than assumed.
The platform's defining question is not "best topology" *or* "best model" in The platform's defining question is not "best topology" *or* "best model" in
isolation but their **interaction**. We crossed two topologies (**pipeline**, 3 isolation but their **interaction**. We crossed two topologies (**pipeline**, 3
stages; **debate**, propose→critique→revise→judge) with three model casts stages; **debate**, propose→critique→revise→judge) with four homogeneous/mixed
(homogeneous Claude, homogeneous GLM‑4.7, and a heterogeneous GLM→Kimi→Claude model casts (**Claude**, **GLM‑4.7**, **Gemini‑2.5‑flash**, and a heterogeneous
cast) on a fixed decision task — *"Rust or Go for a 3‑person startup backend?"* — and **GLM→Kimi→Claude**) on a fixed decision task — *"Rust or Go for a 3‑person startup
ranked all six final outputs 1–6 with an independent judge on reasoning quality, backend?"* — and ranked all eight final outputs 1–8 with an independent judge on
clarity, and concision: reasoning quality, clarity, and concision:
| Topology \ cast | Claude | GLM‑4.7 | Heterogeneous | | Topology \ cast | Claude | GLM‑4.7 | Gemini | Heterogeneous |
|------------------|:------:|:-------:|:-------------:| |------------------|:------:|:-------:|:------:|:-------------:|
| **Pipeline** | 3 | **2** | 4 | | **Pipeline** | 4 | **2** | 6 | 3 |
| **Debate** | **6** | **1** | 5 | | **Debate** | **8** | **1** | 7 | 5 |
Two results stand out. First, **all six cells converged on the same decision** ("Go") Three results stand out. First, **all eight cells converged on the same decision**
— so on a task with a clear prior, topology and model move *justification quality*, ("Go") — so on a task with a clear prior, topology and model move *justification
not the answer. Second, and central: **the topology × model interaction is real and quality*, not the answer. Second, and central: **the topology × model interaction is
non‑monotone.** Debate *amplified* the strongest model (GLM‑4.7: pipeline 2nd → real and non‑monotone.** Debate *amplified* the single strongest model (GLM‑4.7:
debate 1st, its extra adversarial round adding a hiring‑cost nuance) but *degraded* pipeline 2nd → debate 1st, its extra adversarial round adding a hiring‑cost nuance)
the others (Claude: 3rd → dead last, its judge wasting words on an "upstream agents but *degraded* every other cast (Claude: 4th → **dead last**, its judge wasting words
agree" meta‑citation; heterogeneous: 4th → 5th). In other words, **the best topology on an "upstream agents agree" meta‑citation; Gemini: 6th → 7th; heterogeneous: 3rd →
depends on the model, and vice‑versa** — debate is not a universal upgrade; it pays 5th). **Debate is not a universal upgrade — it pays off only with a model strong
off only with a model strong enough to use the extra rounds. This is precisely the enough to exploit the extra rounds, and is actively harmful otherwise.** Third,
result the platform is built to produce and that neither a topology‑only nor a **model strength is task‑dependent, not absolute**: Gemini‑2.5‑flash placed *2nd* on
model‑only study can see: the optimal *configuration* is a joint choice over the §7.3 concision brief but *6th–7th* here — the judge faulted it for a "generic
{topology × per‑role model}, discovered empirically, with the §5 safety invariant list" and "the same idea repeated," i.e. concise‑but‑shallow, an asset for brevity
holding across every cell. The sweep was run sequentially under the subscription and a liability for nuanced reasoning. The same model is strong or weak depending on
plans' quota/concurrency caps (21 turns total), the practical envelope for this the role's rubric. Together these confirm the result the platform is built to produce
class of experiment. and that neither a topology‑only nor a model‑only study can see: the optimal
*configuration* is a joint choice over {topology × per‑role model × task}, discovered
empirically, with the §5 safety invariant holding across every cell. The sweep ran
sequentially under the subscription plans' quota/concurrency caps, the practical
envelope for this class of experiment.
## 8. Limitations and future work ## 8. Limitations and future work