MSQA: A Natively Sourced Multilingual and Multicultural SimpleQA Benchmark
2026-07-01 • Computation and Language
Computation and Language
AI summaryⓘ
The authors studied whether language models that speak multiple languages also understand the cultures behind those languages, which they call the Illusion of Cultural Alignment. They created a new test with real, culture-specific questions in 11 languages to check this. After testing 18 models, they found that cultural understanding mainly depends on how much the model saw that culture during training, not just its overall smarts. Common tricks used during testing didn’t fix this gap, meaning models can be fluent but still misunderstand cultural context. The authors conclude that just knowing a language doesn’t mean the model truly grasps the culture without more thorough solutions.
Multilingual fluencyCultural alignmentLarge Language Models (LLMs)Cross-lingual transferCultural competencePre-training exposureBenchmark datasetInference-time remediesRetrieval augmentationLocality effect
Authors
Xianru Chen, Yukai Huang, Mingxiang Chen, Xinping Lei, Fangbing Deng, Jin Chen, Ge Zhang, Wenhao Huang, Jiaheng Liu
Abstract
Multilingual fluency often invites a stronger assumption: a model that can speak a user's language must also understand the culture encoded by that language. We call this the Illusion of Cultural Alignment. To test this assumption directly, we introduce MSQA, a benchmark of 1,064 natively sourced questions across 11 language groups, five cultural dimensions, and three difficulty tiers. Unlike translated benchmarks, MSQA targets locally grounded knowledge and reduces shortcuts from English-centric cross-lingual transfer. Evaluating 18 LLMs, we find substantial cultural degradation and a pronounced Locality Effect: cultural competence tracks pre-training exposure more closely than general reasoning ability. We further show that common inference-time remedies do not dissolve the illusion. Models remain overconfident on unfamiliar cultural questions, repeated sampling yields unstable rather than reliable correctness, and retrieval augmentation helps unevenly on long-tail facts. These findings indicate that cultural alignment cannot be inferred from multilingual ability alone and requires deeper intervention than calibration, sampling, or retrieval at inference time