AI summaryⓘ
The authors studied how to make conversational search systems, which find documents based on chat history, more efficient without losing accuracy. They focused on using a method called knowledge distillation with Kullback-Leibler Divergence (KLD) to teach smaller models to rewrite queries like large language models do. They found adding a contrastive loss improved search precision and that the way samples are chosen for training affects performance. They also addressed efficiency problems caused by longer conversations by applying a regularization technique that made the model's output sparser, doubling speed with minimal loss in accuracy. Their work offers practical advice to balance effectiveness and efficiency in conversational search retrieval models.
Conversational SearchKnowledge DistillationLarge Language ModelsQuery RewritingKullback-Leibler DivergenceContrastive LossSparse RetrievalRegularizationRecall@100Efficiency in Inference
Authors
Stan Fris, Jan Hutter, Jan Henrik Bertrand, Simon Lupart, Mohammad Aliannejadi
Abstract
Conversational Search (CS) considers retrieval of relevant documents based on conversational context. Large Language Models (LLMs) have significantly enhanced CS by enabling effective query rewriting. However, employing LLMs during inference poses efficiency challenges. A method to balance effectiveness and efficiency is the use of knowledge distillation from LLM-based query rewriting. Recent work applies the Kullback-Leibler Divergence (KLD) for distillation, relaxing the alignment with the teacher signal compared to previous methods. Despite these gains, several aspects of KLD-based distillation for conversational search remain understudied, and we investigate them in this work. Prior work in related fields suggests that adding a contrastive loss to the KLD objective can improve performance; we confirm this and observe significant gains in precision-oriented ranking metrics. We also find that contrastive sampling strategies for the KLD loss have a non-trivial impact and must be chosen carefully. Although theory suggests that more samples improve the KLD estimate, experiments show diminishing returns on the number of used samples. Finally, we address the phenomenon of decreased sparsity in longer conversations, which limits computational efficiency across sparse retrieval methods. We find that the representations from the model distilled with the KLD loss can be strongly regularized with a regularization loss, substantially improving sparsity and inference efficiency without significantly harming retrieval effectiveness. We achieve a $2\times$ decrease in FLOPS on TopiOCQA with negligible loss in effectiveness, corresponding to a $\leq 2%$ drop in Recall@100. Our results provide insights into distillation objectives for learned sparse conversational retrievers and offer practical guidelines for improving effectiveness and efficiency in first-stage retrieval.