AI Overviews in Academic Search: Evaluating AI-generated Summaries of Search Results in a Domain-specific Search Engine
2026-07-03 • Information Retrieval
Information RetrievalHuman-Computer Interaction
AI summaryⓘ
The authors studied whether AI-generated summaries of search engine results help people find social science research more easily. They created a list of common errors in these summaries and tested how users felt and behaved with and without them. While users generally found the summaries somewhat helpful and less frustrating, the AI summaries did not significantly improve their performance or satisfaction. The authors conclude that AI summaries can be a useful tool depending on the user and context, but they are not a guaranteed improvement. They also offer advice on safely using AI summaries in academic search environments.
Search Engine Results Pages (SERPs)AI-generated summariesAcademic search enginesUser studyInformation Foraging TheoryError taxonomyWorkloadUsabilityResult relevanceQuery reformulation
Authors
Kevin Schott, Kanishka Silva, Ingo Frommholz, Philipp Mayr, Dagmar Kern, Daniel Hienert
Abstract
Evaluating search engine results pages (SERPs) to assess result relevance is a demanding step in academic search. In a formative mixed-methods design study, we examine AI-generated SERP-level summaries as a support feature in an academic search engine for social science information. First, we manually evaluated summaries of the top five results for 10 queries using two general-purpose models, one commercial and one open, deriving an exploratory six-category error taxonomy and five safeguards for scholarly deployment. We then conducted a within-subjects user study (n = 30) comparing interfaces with and without AI summaries. Confirmatory analyses showed consistent but non-significant trends favoring AI summaries for subjective workload, perceived usefulness, satisfaction, and decision-making confidence. Exploratory analyses suggested lower mental demand, with frustration also tending to be lower. Behaviorally, participants rarely expanded the summaries and descriptively made slightly fewer result clicks and query reformulations when summaries were available. Drawing on Information Foraging Theory and participant feedback, we suggest that AI summaries may concentrate SERP-level information scent to support early triage. Overall, the findings indicate that SERP-level AI summaries are a context- and user-dependent aid rather than a universal improvement, while contributing an error taxonomy, safeguard-aware deployment guidance, and concrete design implications for scholarly search.