Prompt-Model Interaction Reaches the Fixed Points: A deterministic, task-free structural readout -- and the factorizations of it that failed
2026-08-21 • Computation and Language
Computation and Language
AI summaryⓘ
The authors studied how prompts interact with language models by looking at a simple mathematical property inside the models, unrelated to task performance. They found that the effect of a prompt depends strongly on the specific model and the prompt together, rather than just the prompt alone. Common explanations like prompt length or types of content didn’t explain the differences, and the behavior only showed up with real text, not random inputs. Overall, they argue that understanding prompt effects needs to focus on the specific model-prompt pair rather than general rules.
prompt engineeringlanguage modelsfixed-point structureargmax mapinstruction tuningconditional distributionattention mechanismsmodel interpretabilitybidirectionalitytask accuracy
Authors
Nicolás Vera Zúñiga
Abstract
That a prompt's effect is not a property of the prompt is established: prompts optimised for one model degrade on another, and rankings reorder under neutral reformatting. That evidence is about task accuracy, which cannot say whether the interaction is a fact about task machinery or about the conditional distribution itself. We ask on a readout with no task in it: the fixed-point structure of the short-window argmax map x_{t+1} = argmax_x p(x | x_{t-1}, x_t), censused from 96 starts. It is deterministic, so nothing can be helped or hurt, and it exists only at short windows -- four of six models lose it entirely by window 16 -- so everything here concerns how a model reads a fragment. Two results. First, the interaction reaches this readout at full magnitude: nine tokens of conditioning move the fixed-point fraction across most of its range, change a four-way structural class, and reorder models, while instruction tuning worth 60.5 IFEval points moves the class by zero. Second, nothing we proposed carries it. Prefix length fails: the effect is not monotone. Four phenomenological factors -- prose-versus-markup, a universal direction, bidirectionality, instruct-resistance -- were each withdrawn within one run of being proposed, dissolved by widening the sample. And the nearest mechanistic account, attention-sink dominance of early tokens, predicts the sign of the shift on 2 of 5 models -- chance -- while a length-by-content cross shows it holds on real text and fails on our probe's uniformly random input, so we are outside its regime, not against it. One fixed nine-token prefix drives four models toward 0 and two toward 1; the bidirectionality survives in-distribution starts. On this readout the unit of explanation is the prompt-model pair. The recurring error it caught in us has a name: a criterion with a shape applied to a quantity with no room to vary.