HETERQA: Benchmarking Record Retrieval over Multiple Heterogeneous Sources
2026-07-03 • Information Retrieval
Information Retrieval
AI summaryⓘ
The authors created HETERQA, a new test set with 857 question-answer pairs designed to help computers find information from five different types of data sources together, like business details from Yelp combined with text and images. They built it by carefully checking and combining data before pairing it with natural language questions. When they tested different computer methods to find answers, none performed perfectly, showing that HETERQA is a hard challenge. This benchmark helps researchers improve ways to search across varied data types. They also shared the dataset and code for others to use.
heterogeneous dataquestion answering (QA)record retrievalbenchmark datasetYelp business recordsRecall@10MRR@10sparse retrievaldense retrievalhybrid retrieval
Authors
Yaodong Su, Hanchang Li, Quanqing Xu, Chuanhui Yang, Yixiang Fang
Abstract
In emerging systems (e.g., social media and e-commerce platforms), data records are often drawn from heterogeneous sources, such as relational tables, text documents, image repositories, spatial databases, and knowledge graphs. Accordingly, retrieving target records for question-answering (QA) tasks requires us to jointly exploit these heterogeneous sources. However, most existing benchmarks are constructed from individual sources, and only a very few recent benchmarks have considered two or three sources. To alleviate this issue, we introduce HETERQA, a comprehensive benchmark with 857 QA pairs for record retrieval over five heterogeneous sources. HETERQA instantiates this setting with Yelp business records, each of which is grounded by multiple sources. We build HETERQA in an answer-driven manner: candidate records are first initialized with record-field constraints, then enriched through heterogeneous sources, and finally cross-verified across required sources before the natural-language question is retained. We validate the benchmark through contradiction detection and human validation, and further evaluate sparse, dense, hybrid, late-interaction, and agentic retrievers under the same metrics. The results show that HETERQA is challenging: hybrid retrieval achieves the strongest Recall@10, Self-RAG achieves the best MRR@10, and all evaluated methods remain far from saturating the benchmark. These findings indicate that HETERQA provides an effective testbed for record retrieval over heterogeneous sources and leaves substantial room for future retrieval methods. The benchmark dataset and source code are publicly available at https://huggingface.co/datasets/hanchang02/HeterQA and https://github.com/hanchang02/HeterQA, respectively.