AI summaryⓘ
The authors identify that previous methods for evolving prompts in NLP tasks tend to make prompts much longer without improving accuracy, a problem they call "prompt bloat." They introduce ESPO, a new approach that breaks prompt optimization into three steps: finding error patterns, generating diverse candidates, and selecting stable prompts using a statistical method. Tested on seven NLP benchmarks, ESPO achieves better accuracy than the previous best method while producing significantly shorter prompts and faster results. Experiments with different language models confirmed ESPO's consistent improvements. They also show that diversity in candidates only helps when combined with their selection strategy.
prompt optimizationevolutionary algorithmserror clusteringbootstrap stability selectionnatural language processing (NLP)benchmark datasetsmodel generalizationcandidate generationprompt bloatcross-model evaluation
Authors
Lihao Liu, Peng Tang, Kunwar Yashraj Singh, Shabnam Ghadar
Abstract
Evolutionary prompt optimizers such as GEPA suffer from prompt bloat: each iteration appends rules and caveats, producing prompts up to 3$\times$ longer yet no more accurate. We trace this to three deficiencies - incomplete error observation, limited search diversity, and unreliable selection - and propose ESPO (Error-Structured Prompt Optimization), which decomposes prompt optimization into three phases: Diagnose clusters all training errors into structural patterns in one round; Propose generates candidates via four complementary strategies with independent biases; Select applies bootstrap stability selection. On seven public NLP benchmarks - Tweet, MMLU, GSM8K, HotpotQA, ScoNe, HoVer, and PUPA - ESPO improves average accuracy by $+$3.76 pp over the state-of-the-art (74.67% vs 70.91% for GEPA), matching or exceeding GEPA on every dataset while producing prompts 47% shorter (1,004 vs 1,878 chars) and faster at inference. Cross-model experiments across four additional student models (Gemma 3 12B, Mistral 14B, Qwen3 32B, Claude Haiku 4.5) show ESPO yields the best average accuracy on every model tested, with the largest gap on Qwen3 GSM8K (15.00% $\to$ 91.40%). A generalization bound (Appendix) grounds each phase in a corresponding term of the test-time gap, and the ablation confirms a key prediction: adding diversity without bootstrap selection actually hurts performance ($-$1.20%).