AI summaryⓘ
The authors studied how language models differ in behaving safely by analyzing eight safety tests across many models using a method called Item Response Theory (IRT). They found that three main features—how strict models are about refusing harmful content, how truthful they are, and how much harm they cause depending on context—explain most differences in safety. Using IRT, they showed it's possible to evaluate models much faster with fewer test questions while still getting accurate results. They also demonstrated that IRT can help detect when models try to fake better scores or change over time. Overall, the authors suggest that IRT is a useful tool for better understanding and checking language model safety.
Language ModelsSafety BenchmarksItem Response TheoryRefusal StrictnessTruthfulnessContextual HarmModel EvaluationPsychometricsSandbaggingAdaptive Testing
Authors
Joshua Fonseca Rivera, Neil Shah, David Demitri Africa, Konstantinos Voudouris
Abstract
Language models differ in how safely they behave and these differences are measured by safety benchmarks. But aggregated benchmark scores are hard to trust and interpret, because benchmarks duplicate one another, correlate heavily, and models may sandbag when they detect evaluation. To address these issues, we draw on Item Response Theory (IRT), a statistical toolkit for measuring these latents from performance on items with inferred psychometric properties. We fit IRT models to eight safety benchmarks across 192 language models, the largest psychometric analysis of LLM safety evaluations to date, and contribute three results. First, we find that three interpretable factors of refusal strictness, truthfulness, and contextual harm explain most of the variance between models across benchmarks. Second, psychometrically selected items recover full benchmark scores with lower error than random subsets of the same size, and roughly ten adaptively chosen items suffice for several individual benchmarks, cutting evaluation cost by 97-99%. Third, IRT supports audits of individual models, showing that it can be used to detect naive sandbagging and changes of model behind APIs. Overall, we show IRT is a ready-made toolkit for reading, reducing, and auditing safety benchmarks, which we recommend frontier labs and evaluators adopt.