AI summaryⓘ
The authors studied how instruction-tuned large language models (LLMs) handle simple restrictions like banning a punctuation mark or common word. They found that these limits make the models produce much less complete answers, losing about 14-48% of important content, and baseline answers are preferred most of the time. This problem appears even in commercial models like GPT-4o-mini, which shows a 31% drop in completeness. The authors traced this fragility to a planning issue caused by instruction tuning, as base models without such tuning do not exhibit similar collapses. They also discovered that usual evaluation methods miss much of this quality loss, revealing a gap in how constrained LLM outputs are assessed.
instruction tuninglarge language modelslexical constraintsresponse comprehensivenessplanning failureclosed-weight modelsGPT-4o-minipairwise evaluationrepresentation analysisLLM evaluation methods
Authors
Erfan Baghaei Potraghloo, Seyedarmin Azizi, Souvik Kundu, Massoud Pedram
Abstract
Instruction-tuned large language models produce helpful, structured responses, but how robust is this helpfulness when trivially constrained? We show that simple lexical constraints (banning a single punctuation character or common word) cause instruction-tuned LLMs to collapse their responses, losing 14--48% of comprehensiveness in pairwise evaluation across three open-weight model families and one closed-weight model (GPT-4o-mini). The baseline response is preferred in 77--100% of 1,920 pairwise comparisons judged by GPT-4o-mini and GPT-4o. Notably, GPT-4o-mini suffers 31% comprehensiveness loss (99% baseline win rate), demonstrating that the fragility extends to commercially deployed closed-weight models, contrary to prior findings on format-level constraints. Through mechanistic analysis, we identify this as a planning failure: two-pass generation (free generation followed by constrained rewriting) recovers 59--96% of response length, and linear probes on prompt representations predict response length with $R^2 = 0.51$--$0.93$ before generation begins, with $R^2$ tracking collapse severity across models. The same probes yield negative $R^2$ on base models, confirming that instruction tuning creates the representational structure encoding the collapse decision. Crucially, base models show no systematic collapse under identical constraints, with effects that are small, noisy, and bidirectional, demonstrating that instruction tuning creates this fragility by coupling task competence to narrow surface-form templates. The effect replicates on MT-Bench across all eight task categories. We further show that standard independent LLM-as-judge evaluation detects only a 3.5% average quality drop where pairwise evaluation reveals 23%, exposing a methodological blind spot in how constrained generation is assessed.