HalluTruthQA-4K: A Fine-Grained Corpus and Annotation Process for Arabic Hallucination Detection and Truth Verification
2026-08-04 • Computation and Language
Computation and Language
AI summaryⓘ
The authors created HalluTruthQA-4K, a large Arabic dataset to help spot and explain factual mistakes made by language models when answering questions. It includes 4,000 examples from topics like Islamic knowledge, history, science, and geography, with correct answers and carefully marked wrong parts. Unlike earlier datasets, this one highlights exactly what parts of an answer are incorrect and why, with expert explanations. The authors also provide detailed information about how they made and checked the data to ensure high quality. This resource aims to improve tools that detect and understand errors in Arabic AI-generated text.
Large Language ModelsArabic NLPHallucinationFact-checkingDataset AnnotationError LocalizationFactual VerificationQuestion AnsweringInter-annotator AgreementKnowledge-intensive Domains
Authors
Salah Eddine Bekhouche, Abdessalam Bouchekif, Hichem Telli, Mohammed-En-Nadhir Zighem, Abdenour Hadid
Abstract
Large language models can generate fluent Arabic answers while introducing factual errors that are difficult to identify and verify. Existing Arabic hallucination resources often assign a binary label to an entire response, indicating whether it is hallucinated or non-hallucinated, but provide limited information about the exact erroneous content, the reason for the error, or the correct factual answer. We present HalluTruthQA-4K, an expanded version of the HalluTruthQA resource containing 4,000 expert-curated Arabic question-answering instances across four knowledge-intensive domains: Islamic knowledge, history, science, and geography. Serving as the official dataset for Track 2 of the HalluScoring 2026 shared task, HalluTruthQA-4K extends our original corpus to 4,000 instances. Each instance pairs an Arabic question with a model-generated response, a verified reference answer, and five plausible distractors. Hallucinated responses are additionally annotated with character-level erroneous spans, human-written explanations, and hierarchical hallucination types. The corpus contains 1,643 hallucinated and 2,357 non-hallucinated responses, with 1,843 annotated erroneous spans. We describe the resource construction and annotation methodology, including question selection, controlled answer generation, candidate construction, expert annotation, independent verification, adjudication, and quality control. We also document the annotation guidelines, taxonomy, data format, inter-annotator agreement, and corpus statistics. HalluTruthQA-4K provides a reusable resource for hallucination detection, span-level error localization, explanation generation, factual verification, and the broader evaluation of factual reliability in Arabic language models.