paper §7.4: complete 8-cell topology×model grid (Gemini folded in)
Gemini's project-wide key-restriction 403 is cleared (key restricted to the
Generative Language API), so Gemini rejoins the grid. Full 2×4 ranking
(pipeline/debate × Claude/GLM-4.7/Gemini/heterogeneous), judge-ranked 1-8:
D2 > P2 > P3 > P1 > D3 > PG > DG > D1.
Adds a third finding: model strength is task-dependent — Gemini-2.5-flash
placed 2nd on the §7.3 concision brief but 6th-7th here on reasoning
("generic list", "idea repeated"): concise-but-shallow. Reinforces that the
optimal config is a joint choice over {topology × per-role model × task}.
Gemini cells ran via the durable async run path (enqueue → worker → poll).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
e8486ee57c
commit
2fcec622fb
@@ -232,32 +232,36 @@ measured rather than assumed.
|
|||||||
|
|
||||||
The platform's defining question is not "best topology" *or* "best model" in
|
The platform's defining question is not "best topology" *or* "best model" in
|
||||||
isolation but their **interaction**. We crossed two topologies (**pipeline**, 3
|
isolation but their **interaction**. We crossed two topologies (**pipeline**, 3
|
||||||
stages; **debate**, propose→critique→revise→judge) with three model casts
|
stages; **debate**, propose→critique→revise→judge) with four homogeneous/mixed
|
||||||
(homogeneous Claude, homogeneous GLM‑4.7, and a heterogeneous GLM→Kimi→Claude
|
model casts (**Claude**, **GLM‑4.7**, **Gemini‑2.5‑flash**, and a heterogeneous
|
||||||
cast) on a fixed decision task — *"Rust or Go for a 3‑person startup backend?"* — and
|
**GLM→Kimi→Claude**) on a fixed decision task — *"Rust or Go for a 3‑person startup
|
||||||
ranked all six final outputs 1–6 with an independent judge on reasoning quality,
|
backend?"* — and ranked all eight final outputs 1–8 with an independent judge on
|
||||||
clarity, and concision:
|
reasoning quality, clarity, and concision:
|
||||||
|
|
||||||
| Topology \ cast | Claude | GLM‑4.7 | Heterogeneous |
|
| Topology \ cast | Claude | GLM‑4.7 | Gemini | Heterogeneous |
|
||||||
|------------------|:------:|:-------:|:-------------:|
|
|------------------|:------:|:-------:|:------:|:-------------:|
|
||||||
| **Pipeline** | 3 | **2** | 4 |
|
| **Pipeline** | 4 | **2** | 6 | 3 |
|
||||||
| **Debate** | **6** | **1** | 5 |
|
| **Debate** | **8** | **1** | 7 | 5 |
|
||||||
|
|
||||||
Two results stand out. First, **all six cells converged on the same decision** ("Go")
|
Three results stand out. First, **all eight cells converged on the same decision**
|
||||||
— so on a task with a clear prior, topology and model move *justification quality*,
|
("Go") — so on a task with a clear prior, topology and model move *justification
|
||||||
not the answer. Second, and central: **the topology × model interaction is real and
|
quality*, not the answer. Second, and central: **the topology × model interaction is
|
||||||
non‑monotone.** Debate *amplified* the strongest model (GLM‑4.7: pipeline 2nd →
|
real and non‑monotone.** Debate *amplified* the single strongest model (GLM‑4.7:
|
||||||
debate 1st, its extra adversarial round adding a hiring‑cost nuance) but *degraded*
|
pipeline 2nd → debate 1st, its extra adversarial round adding a hiring‑cost nuance)
|
||||||
the others (Claude: 3rd → dead last, its judge wasting words on an "upstream agents
|
but *degraded* every other cast (Claude: 4th → **dead last**, its judge wasting words
|
||||||
agree" meta‑citation; heterogeneous: 4th → 5th). In other words, **the best topology
|
on an "upstream agents agree" meta‑citation; Gemini: 6th → 7th; heterogeneous: 3rd →
|
||||||
depends on the model, and vice‑versa** — debate is not a universal upgrade; it pays
|
5th). **Debate is not a universal upgrade — it pays off only with a model strong
|
||||||
off only with a model strong enough to use the extra rounds. This is precisely the
|
enough to exploit the extra rounds, and is actively harmful otherwise.** Third,
|
||||||
result the platform is built to produce and that neither a topology‑only nor a
|
**model strength is task‑dependent, not absolute**: Gemini‑2.5‑flash placed *2nd* on
|
||||||
model‑only study can see: the optimal *configuration* is a joint choice over
|
the §7.3 concision brief but *6th–7th* here — the judge faulted it for a "generic
|
||||||
{topology × per‑role model}, discovered empirically, with the §5 safety invariant
|
list" and "the same idea repeated," i.e. concise‑but‑shallow, an asset for brevity
|
||||||
holding across every cell. The sweep was run sequentially under the subscription
|
and a liability for nuanced reasoning. The same model is strong or weak depending on
|
||||||
plans' quota/concurrency caps (21 turns total), the practical envelope for this
|
the role's rubric. Together these confirm the result the platform is built to produce
|
||||||
class of experiment.
|
and that neither a topology‑only nor a model‑only study can see: the optimal
|
||||||
|
*configuration* is a joint choice over {topology × per‑role model × task}, discovered
|
||||||
|
empirically, with the §5 safety invariant holding across every cell. The sweep ran
|
||||||
|
sequentially under the subscription plans' quota/concurrency caps, the practical
|
||||||
|
envelope for this class of experiment.
|
||||||
|
|
||||||
## 8. Limitations and future work
|
## 8. Limitations and future work
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user