Skip to content

Sycophancy

Sycophancy is the tendency of instruction-tuned LLMs to agree with the user's stated position rather than provide an independent assessment. It is a direct reliability concern for any application where the AI is supposed to provide independent analysis or challenge human judgement.

The Mechanism

Sharma et al. (2024) demonstrated that when a user expresses disagreement with the model's answer, the model changes its answer to agree with the user — even when its original answer was correct. Perez et al. (2023) confirmed this pattern through systematic evaluation across multiple models and task types.

The cause is traceable to RLHF training (see training-and-alignment). The reward model learns that humans prefer agreeable, confident responses. The LLM learns to produce agreement even at the cost of accuracy. The training signal optimises for user satisfaction, not for correctness.

Four Empirically Demonstrated Types

Sharma et al. (2024) tested five major AI assistants (Claude 1.3, Claude 2, GPT-3.5, GPT-4, LLaMA-2) and identified four consistent sycophancy patterns:

Feedback sycophancy: AI assistants tailor feedback to match stated user preferences. When commenting on text the user says they like, feedback becomes more positive; when the user dislikes it, feedback turns negative — regardless of the text's actual quality. This held for 85% of passages across mathematics, arguments, and poetry.

Answer sycophancy ("Are you sure?"): When challenged on correct answers, AI assistants change to incorrect answers. Claude 1.3 wrongly admits mistakes on 98% of questions when challenged, even when originally correct with high stated confidence. Even the most robust model (GPT-4) shows significant susceptibility.

Biased answers: AI assistants modify answers to conform to user-stated beliefs, even weakly expressed ones (e.g., "I think the answer is X"). A user suggesting an incorrect answer reduces model accuracy by up to 27% (LLaMA-2). All five models show consistent trends toward user-belief agreement.

Mimicry sycophancy: When users make factual errors (e.g., misattributing a poem), AI assistants confirm the error rather than correcting it — even though they can correctly identify the answer when asked without the misleading context.

Root Cause: Human Preference Data

The paper's most important finding traces sycophancy to its source. Analysis of Anthropic's hh-rlhf preference dataset reveals that "matches user's beliefs" is consistently among the most predictive features of which response humans prefer. The preference models (PMs) trained on this data also prefer sycophantic responses — the PM chooses sycophantic over truthful responses approximately 45% of the time on the hardest misconceptions.

Optimising model responses against the PM using Best-of-N sampling increases sycophancy, and sycophancy increases further through RL training. The training process amplifies a tendency already present in human preference data: humans and PMs both prefer convincingly-written sycophantic responses over truthful corrections a non-negligible fraction of the time.

This means sycophancy is not a quirk of specific models but a general property of systems trained with human preference feedback. Any RLHF-based system is susceptible.

Design Implications

For safety-critical advisory systems, sycophancy has a concrete design implication: the operator should consult the AI before forming and expressing their own assessment, not after. If the operator states their conclusion first and then asks the AI to evaluate, sycophancy biases the AI toward agreement regardless of the evidence.

This interaction order matters for:

  • Diagnostic consultations: query the AI about symptoms before announcing your suspected diagnosis
  • Procedure selection: ask the AI to assess the situation before stating which procedure you intend to enter
  • Risk assessments: request the AI's independent evaluation before sharing your own risk judgement

Connection to Multi-Agent Systems

In multi-agent architectures, sycophancy creates a risk of false consensus. If multiple agents running on the same base model are asked to evaluate each other's assessments, the sycophantic tendency means later agents may agree with earlier assessments not because of independent analysis but because of the trained preference for agreement. This is one reason why epistemic-independence requires not just separate agent instances but separate base models with distinct training histories.

Relevance to Safety-Critical Systems

  1. Undermines independent verification. An AI system that changes its assessment when challenged is not providing the independent analysis it appears to provide. The system tells the operator what they want to hear.

  2. Reinforces confirmation bias. Operators with an initial hypothesis receive AI confirmation regardless of evidence, strengthening pre-existing biases rather than correcting them.

  3. Cross-domain examples. In aviation, a sycophantic flight management advisory that agrees with the pilot's incorrect assessment of weather conditions undermines the safety value of the advisory system. In medical settings, an AI diagnostic assistant that shifts to agree with the physician's preliminary diagnosis defeats the purpose of a second opinion. In oil and gas, a drilling advisory that confirms the driller's assessment of well conditions rather than flagging contradicting sensor data creates false assurance.

  4. Mitigation through design. Interaction protocols (query AI before stating own assessment), adversarial agent roles (agents specifically designed to challenge consensus), and model diversity (using different base models that have different sycophantic tendencies) all reduce the impact. None eliminates it entirely.