AI summaryⓘ
The authors created a special way to test large language models (LLMs) for use in Dutch government settings, considering both local language needs and public administration values. They worked with experts and users to identify six important areas to evaluate: truthfulness, honesty, bias, energy use, cost, and clarity about training data. Testing over 30 different models showed no one model was best at everything, with trade-offs between quality, cost, and environmental impact. They also found that being factually correct and being honest about not knowing something are different and don’t always go together. To help decision-makers, the authors made a simple-to-use overview of these results meant for a wide range of stakeholders.
large language modelsevaluation frameworkpublic administrationfactualityhonestysocial biasenergy consumptiontraining data transparencymultilingual modelsgovernment use
Authors
Laurens Samson, Iva Gornishka, Gossa Lô, Yuki M. Asano, Sennay Ghebreab
Abstract
Large language models are increasingly being deployed in governmental settings, yet few existing evaluation frameworks jointly reflect the values of public administration and the linguistic requirements of non-English contexts. We present the "Grip on LLMs" framework, a systematic evaluation suite for Dutch governmental use developed in collaboration with domain experts from a major Dutch municipal organisation. Through an advisory board process, user research, and a survey of the users of a civil-servant chatbot, we identify six evaluation dimensions (factuality, honesty, social bias, energy consumption, cost, and training data transparency) and operationalise them into a benchmark suite covering more than 30 multilingual and Dutch-specific models. Our results reveal that no single model excels across all dimensions, and that trade-offs are unavoidable: higher quality consistently comes at greater environmental impact and financial cost, while bias remains largely independent of both. We further find that factuality (whether a model answers correctly) and honesty (whether a model acknowledges what it does not know) are governed by distinct properties, with high factuality not implying high honesty. To make these findings actionable for non-technical audiences, we release a publicly accessible, user-friendly model overview designed for the full range of stakeholders involved in governmental LLM selection, from engineers to policymakers.