LLMLingua — Compressing Prompts for Accelerated Inference of Large Language Models¶
Jiang et al. (2023) present LLMLingua, a coarse-to-fine prompt compression method that achieves up to 20x compression of prompts with minimal performance loss (1.5 points on reasoning benchmarks). The method uses a small language model to identify and remove low-information tokens from prompts before they are sent to a larger target LLM, reducing both latency and cost.
Method¶
LLMLingua has three components:
Budget Controller allocates different compression ratios to different parts of the prompt. Instructions and questions receive higher budgets (less compression) because they have the most direct influence on output quality. Demonstrations receive lower budgets because their information is often redundant. The controller uses perplexity scores from a small language model to rank demonstrations by information content, keeping the most informative ones.
Iterative Token-level Prompt Compression (ITPC) compresses the remaining prompt at token granularity. A small language model computes conditional perplexity for each token segment, and tokens with perplexity below a dynamically calculated threshold are removed. The iterative approach processes text in segments, updating conditional probabilities as it goes — this avoids the conditional independence assumption that degrades simpler methods.
Distribution Alignment addresses the gap between the small model used for compression and the large target LLM. The small model is fine-tuned on data generated by the target LLM to align their probability distributions, improving compression quality.
Key Results¶
On GSM8K (mathematical reasoning), LLMLingua achieves 79.08 Exact Match at 5x compression (1-shot constraint) compared to 78.85 with the full prompt. At 14x compression (half-shot), it still achieves 77.41 — retaining reasoning ability despite removing the majority of tokens. On conversation (ShareGPT) and summarisation (Arxiv-March23) tasks, it achieves 4x–9x compression with high BERTScore F1, indicating semantic preservation.
The method achieves 1.7x to 5.7x end-to-end speedup on a V100 GPU, with the compression overhead being small relative to the inference savings. It also works across different target LLMs (GPT-3.5-Turbo and Claude-v1.3), demonstrating generalisability.
Connection to Context Length Degradation¶
LLMLingua's effectiveness connects to Du et al. (2025): if context length alone degrades performance, then compressing prompts does double duty — it reduces cost/latency and may improve output quality by shortening the effective context. The budget controller's strategy of preserving instructions and questions while compressing demonstrations aligns with the finding that the model needs the task-critical information nearby and uncluttered.
Relevance to Safety-Critical Systems¶
For context management in safety-critical applications, LLMLingua demonstrates that prompt content is highly redundant from an information-theoretic perspective — most tokens in natural language prompts contribute little to the model's comprehension. This has two implications: (1) systems that aggressively pack context windows with long documents, transcripts, or procedure texts are paying a latency and potentially quality cost for tokens that carry minimal information; (2) principled compression can maintain task performance while substantially reducing the context window footprint, creating headroom for additional high-value information. The trade-off is that compression introduces a new failure mode — the small compression model may misjudge which tokens are important for a specific safety-critical query, removing information that turns out to be decision-relevant.