Skip to content

Context Length Alone Hurts LLM Performance Despite Perfect Retrieval

Du et al. (2025) demonstrate a previously underappreciated limitation of long-context LLMs: the sheer length of the input degrades reasoning and problem-solving performance, independent of retrieval quality and independent of distraction from irrelevant content. Even when models perfectly retrieve all relevant evidence, longer inputs produce worse answers.

The Core Experiment

The study constructs a controlled benchmark by taking short-context problems (math, QA, coding) and extending them with distraction tokens inserted between the evidence and the question. The key innovation is measuring retrieval accuracy and task accuracy separately, allowing the authors to isolate the effect of context length from the effect of retrieval failure.

Perfect retrieval does not prevent degradation. Llama-3.1-8B Instruct retrieves all evidence with 100% exact match for 970 of 1000 MMLU problems when extended to 30K tokens. Yet its accuracy drops by 24.2% compared to the short-context version of the same problems. The model finds the right information but fails to use it correctly.

Eliminating Alternative Explanations

The authors systematically rule out other causes:

Minimal distraction (whitespace). Replacing essay distractions with whitespace tokens — carrying virtually no semantic information — still produces substantial performance drops. For Llama-3 at 30K tokens with whitespace, performance drops include 48% on VarSum and 30% on GSM8K.

Zero distraction (masking). The most striking experiment: masking all distraction tokens entirely so the model attends only to the evidence and question (with masked tokens creating distance between them). Performance still drops — at least 7.9% for both Llama and Mistral at 30K masked tokens, and up to 50% on HumanEval for Llama-3 compared to only 19.4% with whitespace. The input is functionally identical to the short-context version except for the increased distance between evidence and question.

Position controlled. Moving evidence to the end of the input (right before the question, minimising distance) still produces degradation, ruling out the lost-in-the-middle effect as the sole explanation.

Closed-Source Models

GPT-4o, Claude-3.5-Sonnet, and Gemini-2.0 show substantially less degradation than open-source models, maintaining near-perfect performance on simpler tasks (VarSum) even at 30K tokens. However, they still show consistent degradation on harder tasks (MMLU drops of 38.8% for Claude-3.5 at 30K whitespace tokens). The finding is not limited to weaker models.

Mitigation: Retrieve Then Solve

The paper proposes a simple strategy: prompt the model to first recite the retrieved evidence, then solve the problem using only the recited evidence in a new, shorter prompt. This effectively converts a long-context task into a short-context one. The approach improves Mistral-v0.3-7B by up to 31.2% on GSM8K and consistently improves GPT-4o on the RULER benchmark by up to 4% on top of an already high baseline.

Implications for Safety-Critical Systems

This finding has direct consequences for retrieval-augmented-generation design and context-management-risks:

  1. RAG saturation explained. The common observation that RAG performance saturates or degrades as more documents are added to the context may be partly explained by context length itself, not just retrieval noise. Adding more relevant documents may simultaneously improve evidence coverage and degrade reasoning quality.

  2. Long-context is not a substitute for good retrieval. The push toward ever-larger context windows (128K, 1M tokens) does not eliminate the need for selective, focused retrieval. A model with a 128K context window and all relevant documents loaded will perform worse than the same model with a 4K context window containing only the most relevant passages.

  3. Operator implications. In safety-critical advisory systems, operators or system designers who assume that providing more context is always better may inadvertently degrade system performance. Context window management — deciding what to include and what to exclude — is a design parameter with direct reliability consequences.