Can LLM Agents Really Debate? A Controlled Study of Multi-Agent Debate¶
Wu et al. (2025) provide the first controlled study of multi-agent debate (MAD) using the Knight-Knave-Spy logic puzzle — a domain with verifiable ground truth that enables precise measurement of both debate outcomes and the reasoning processes behind them. The study systematically varies six factors (team size, composition, confidence visibility, debate order, depth, and task difficulty) to isolate what actually drives debate success.
Key Findings¶
Intrinsic reasoning strength dominates. The strongest predictor of debate outcome is the initial accuracy of the participating models (β = 0.600, p < 0.001). Performance is effectively bounded by the strongest reasoner in the team. No amount of structural engineering (debate depth, ordering, confidence visibility) compensates for weak base models. This directly challenges claims that multi-agent debate can reliably improve on single-model performance through coordination alone.
Group diversity provides genuine but bounded gains. Heterogeneous teams (mixing models of different strengths) consistently outperform homogeneous teams. Het-Mix D (strong, diverse) achieves the highest accuracy, while Hom-Mix Weak (all weak models) shows negligible benefit from debate. Diversity matters — but it cannot substitute for capability.
Majority pressure suppresses independent correction. Agents within a majority position achieve 97% correction rates (trivially, since they start correct). But agents in the minority — who might hold the correct answer against an incorrect majority — show dramatically lower correction rates (as low as 3.6% for weak models). Weak agents defer to the majority rather than defending correct minority positions, a direct analogue of sycophancy at the multi-agent level.
Structural parameters have limited effect. Debate depth, confidence visibility, and debate order showed no statistically significant effect on final accuracy. This is notable: adding more debate rounds or making confidence visible does not improve outcomes — only the quality and diversity of the participating models matter.
Rational reasoning predicts success. Process-level analysis revealed that debates where agents engaged in validity-aligned reasoning (evaluating evidence, responding to counterarguments) outperformed debates characterised by irrational behaviour (ignoring contradictory evidence, deferring without evaluation). Three desiderata for effective debate: inclusive deliberation, rationale over assertion, and advancement of understanding.
Strong model paradox. When strong models provide high-quality reasoning in heterogeneous teams, weaker teammates paradoxically achieve lower correction rates (80% vs 93% for weak-only teams). Strong reasoning is too sophisticated for weak models to evaluate properly, causing them to defer rather than engage critically.
Relevance to This Wiki¶
This paper provides the most rigorous evidence to date on the conditions under which multi-agent debate can — and cannot — provide genuine epistemic-independence. The finding that majority pressure suppresses minority-correct positions directly connects to multi-agent-coordination-failures and validates concerns about monoculture-collapse: homogeneous teams converge toward shared errors without self-correction. The structural-parameters-don't-matter finding has practical implications for system design: investing in model diversity (different base models with different training) is more valuable than investing in elaborate debate protocols. For safety-critical applications, this means that multi-agent verification architectures should prioritise genuine model heterogeneity over sophisticated coordination mechanisms.