Cognitive Episodes in LLM Reasoning Traces Enable Interpretable Human Item Difficulty Prediction
2026-06-26 • Computation and Language
Computation and LanguageArtificial IntelligenceComputers and SocietyMachine Learning
AI summaryⓘ
The authors suggest that predicting how hard a test question is should consider not just the question text but also how people solve it step-by-step. They use large reasoning models to track these problem-solving steps and group them into meaningful stages, called episodes. Their method, Epi2Diff, uses these episodes along with the question's content to better predict how difficult items are for humans. Tests on real data show their approach works better than previous methods. They also found that harder questions involve more complex and repeated problem-solving steps, not just longer answers.
human item difficultyeducational assessmentlarge reasoning modelsreasoning tracesepisode sequencesproblem-solving statessemantic item representationsSATlanguage model fine-tuningin-context learning
Authors
Chenguang Wang, Ming Li, Xinyue Zeng, Zhuochun Li, Hong Jiao, Tianyi Zhou, Dawei Zhou
Abstract
Predicting human item difficulty is central to educational assessment, where reliable estimates support fairness and effective test construction. Existing methods often depend on costly human calibration or item-level textual representations, providing limited evidence about the cognitive processes that make items difficult. We argue that difficulty should be viewed not only as a property of item text, but also as an observable consequence of the problem-solving burden an item induces. Large Reasoning Models (LRMs) offer scalable process evidence through reasoning traces, but such evidence must be structured to support interpretable modeling. To this end, we introduce Epi2Diff (Episode to Difficulty), a framework that maps LRM reasoning traces into cognitively grounded episode sequences. These episodes group trace segments into functional problem-solving states, enabling difficulty to be modeled through reasoning scale, effort allocation, and state transitions. Epi2Diff extracts compact episode-dynamic features and combines them with semantic item representations for human difficulty prediction. Experiments on four real-world human difficulty datasets show that Epi2Diff consistently outperforms strong baselines, including fine-tuned small language models, LLM in-context learning, and supervised LLM adaptation. On SAT-derived classification benchmarks, Epi2Diff achieves an 8.1% average relative gain over supervised LLM fine-tuning baselines. Further analyses show that harder items induce more effortful, iterative, and implementation-centered episode dynamics, rather than merely longer responses. These results demonstrate that cognitive episodes in LRM reasoning traces provide a predictive and interpretable process representation for human item difficulty, offering a new lens for educational measurement with reasoning models.