AtomEval: Atomic Evaluation of Adversarial Claims in Fact Verification
2026-04-09 • Computation and Language
Computation and LanguageArtificial Intelligence
AI summaryⓘ
The authors looked at how people test fact-checking systems by changing claims to trick them, but found that usual scoring methods miss when these changed claims actually lose their truth. They made a new system called AtomEval that breaks claims into smaller parts and checks if these parts stay true, which helps spot fakes better than before. Testing on a big dataset, their method gave clearer results on how good these tricky claims really are. They also found that stronger AI models don't always create better trick claims when checked carefully, showing current testing isn't perfect.
Adversarial claim rewritingFact-checking systemsFEVER datasetLarge language models (LLMs)Truth-conditional consistencySubject-relation-object-modifier (SROM)Validity-aware evaluationAtomic Validity Scoring (AVS)Adversarial evaluationSemantic corruption
Authors
Hongyi Cen, Mingxin Wang, Yule Liu, Jingyi Zheng, Hanze Jia, Tan Tang, Yingcai Wu
Abstract
Adversarial claim rewriting is widely used to test fact-checking systems, but standard metrics fail to capture truth-conditional consistency and often label semantically corrupted rewrites as successful. We introduce AtomEval, a validity-aware evaluation framework that decomposes claims into subject-relation-object-modifier (SROM) atoms and scores adversarial rewrites with Atomic Validity Scoring (AVS), enabling detection of factual corruption beyond surface similarity. Experiments on the FEVER dataset across representative attack strategies and LLM generators show that AtomEval provides more reliable evaluation signals in our experiments. Using AtomEval, we further analyze LLM-based adversarial generators and observe that stronger models do not necessarily produce more effective adversarial claims under validity-aware evaluation, highlighting previously overlooked limitations in current adversarial evaluation practices.