AI summaryⓘ
The authors introduce the IOL-AI Challenge, a competition using unseen linguistic puzzles from the International Linguistics Olympiad to test AI reasoning without provided rules. They had 46 teams submit solutions under limited computing power and also tested 15 top AI models without such limits. Their results show that smaller models sometimes do better than bigger ones, and improvements come more from how answers are produced than from bigger model size. The study also found that automatic scoring matches human judges well but gives weaker models extra credit and stronger models less. Overall, linguistic reasoning remains a tough test for AI, as prior knowledge about languages doesn’t give a clear advantage.
Large Language Models (LLMs)International Linguistics Olympiad (IOL)linguistic reasoningAI benchmarkingautomatic metricsmodel capacitydecoding strategiescompute budgetjury evaluationgeneralizable reasoning
Authors
Eduardo Sánchez, Rita Berrada, Dan-Mircea Mirea, Sara Rajaee, Alexander Piperski, Ana Meta Dolinar, Boris Iomdin, Andrey Nikulin, Mariya Shmatova, Marzieh Fadaee, Julia Kreutzer
Abstract
Reasoning in LLMs is overwhelmingly studied in domains that provide a model with rules: mathematics and code. Linguistic puzzles invert this: the solver must first discover the system before reasoning within it. We present the IOL-AI Challenge, an open-science competition run on the unseen problems of the International Linguistics Olympiad (IOL) 2026 Individual Contest, evaluated both automatically and, for the first time, by members of the official IOL Jury under the same rubrics applied to human contestants. The challenge drew 731 submissions from 46 teams under a strict compute budget (one T4, 30 mins). We additionally benchmark 15 unconstrained frontier and open models, with Claude Opus 4.8 earning a jury score equivalent to a gold medal, while both resource-constrained systems we submitted for jury grading scored in the range of the bottom 5% of contestants. Capability was not determined by scale: 14B submissions outperform models twice their size, and gains come from decoding and output-handling rather than model capacity. We also found that automatic metrics rank systems exactly as the jury does, but compress the scale, upscoring weak systems by ~13 points and understating strong ones. Our analysis shows that while frontier models might have prior knowledge about some of the problem languages, it does not significantly help them solve the linguistic reasoning tasks, leaving linguistic reasoning as a strong benchmarking proxy for generalizable reasoning skills.