Beyond Document Grounding: Span-Level Hallucination Detection over Code, Tool Output, and Documents
2026-07-01 • Computation and Language
Computation and Language
AI summaryⓘ
The authors created a new test to check how well computer programs can spot made-up or incorrect details (hallucinations) in different kinds of information, like code, tool outputs, and documents. They made the test by taking correct pieces and adding small, clearly labeled fake parts. Their specially trained model, based on Qwen3.5-2B, did much better at finding these fake parts than other methods, especially in code-related tests. It also worked well on regular text-based tests. This helps improve the ability to detect errors in systems that generate content using various data types.
hallucination detectionretrieval-augmented generationstructured inputssource codespan-level detectionQwen3.5-2Bbenchmark datasetRAGTruthPsiloQAzero-shot evaluation
Authors
Ádám Kovács, Bowei He, Xue Liu, István Boros, Szilveszter Tóth, Gábor Recski
Abstract
Hallucination detection for retrieval-augmented generation (RAG) is usually evaluated on natural-language document evidence. However, grounded generation systems increasingly rely on structured inputs: source code, developer-tool output, markdown documents, tables, and repository metadata. We introduce a unified benchmark for span-level hallucination detection over code, tool output, structured documents, and existing natural-language RAG datasets. The benchmark is built by starting from grounded correct answers, injecting localized hallucinations with exact character labels, and validating the code test split with evidence-based review. Our fine-tuned Qwen3.5-2B detector reaches 0.689 span-F1 on the unified test set and 0.60 on the code-agent source, where it substantially outperforms LettuceDetect-large (0.17) and the strongest zero-shot LLM judges we evaluated (at most 0.22). The same model remains competitive on established natural-language benchmarks, with 81.8 RAGTruth example-F1 and 0.724 English PsiloQA IoU.