LittleLearner: Language Models Under Pedagogically Controlled Knowledge Exposure
2026-08-13 • Computation and Language
Computation and LanguageArtificial IntelligenceMachine Learning
AI summaryⓘ
The authors created a special language model called LITTLELEARNER by training it only on text suitable for U.S. elementary school students up to Grade 5. This controlled setup helps researchers understand what the model knows and can do without outside information. They also made a dataset, LITTLECURRICULUM, to support this approach. Their tests show that while the model can learn new things after training, it doesn’t exceed its original knowledge limits. The work provides a simple and clear way to study how language models learn within strict boundaries.
language modelpretraining corpuselementary educationcurriculum guidelinesopen-ended evaluationpost-training learningin-context learningmodel capabilitiestraining data scopeknowledge acquisition
Authors
Fanfei Li, Jana Zeller, Manuel Prada-Corral, Thaddäus Wiedemer, Prasanna Mayilvahanan, Ryan Cotterell, Wieland Brendel
Abstract
Modern language models are trained on heterogeneous web-scale text corpora. Consequently, studying knowledge and skill acquisition is difficult, as prior exposure to related content is hard to characterize. To address this challenge, we introduce LITTLECURRICULUM, a curated 88B-token pretraining corpus tailored to U.S. elementary school material, explicitly excluding concepts, facts, and vocabulary taught above Grade 5. Training a 5B-parameter LLM from scratch on LITTLECURRICULUM yields LITTLELEARNER, a model with sufficient language competence for open-ended evaluation, yet with clear knowledge and capability boundaries mapped to interpretable curriculum guidelines. We release LITTLECURRICULUM and LITTLELEARNER as a developmentally restricted sandbox to study how models acquire, represent, and use data under a well-defined training scope. We illustrate the sandbox's utility in a first suite of experiments on injecting new knowledge through post-training and in-context learning. These methods let LITTLELEARNER better utilize existing knowledge, but do not raise out-of-scope capabilities. Our findings underscore the value of this controlled environment for future investigations.