AI summaryⓘ
The authors studied whether the text produced by a language model can reveal clues about the internal steps the model actually took to generate that text. They tested two types of models on an arithmetic task designed to let the model use different internal processes but still produce the same answer. By enforcing these internal steps and linking them to subtle changes in the text, the researchers were able to detect which internal state was used just from the output. Their results show it is possible, under controlled conditions, to verify some internal computation from the model’s words even when the final answer remains the same. However, in a separate test where state information was not deliberately encoded, they could not find naturally happening intermediate states.
language modelinternal statecomputational provenancetransformerfeed-forward neural networkarithmetic taskcausal computationintermediate statelinear probesstatistical pattern detection
Abstract
A language model's output does not by itself provide verifiable evidence about the internal computation that produced it. We study computational provenance: whether generated text can carry detectable evidence of which causally relevant internal state occurred. We test a bounded form of this idea in two controlled architectures: a modular feed-forward neural network and a transformer-based model. Both architectures are trained on the same arithmetic task with a mandatory pathway through two discrete intermediate states, allowing different internal paths to produce the same answer. We deliberately switch between these paths, authenticate the state actually used, and let that verified state determine a subtle statistical pattern in the generated text that can later be detected. The feed-forward and transformer systems each passed all 128 matched pairs in both their public and separately sealed protected end-to-end evaluations, with the detector recovering the signal associated with the authenticated internal state. The required causal computation also reproduced across five independently trained feed-forward models and three independently trained transformers. In a separate answer-only transformer experiment, our linear probes did not recover a naturally learned intermediate state. These results provide a controlled proof of concept that information about a verified, causally relevant internal state can be preserved in generated text even when the answer is unchanged.