From Threads to Trajectories: A Multi-LLM Pipeline for Community Knowledge Extraction from GitHub Issue Discussions
2026-04-28 • Software Engineering
Software Engineering
AI summaryⓘ
The authors created SWE-MIMIC-Bench, a dataset that organizes long and messy GitHub issue discussions into clear story-like summaries. They use a chain of five different language AI models to break down comments, understand linked resources, label parts of the conversation (like causes or solutions), and then piece everything together coherently. This helps developers and researchers quickly grasp complex problem-solving conversations and can also train AI to act like expert developers. The system was tested on 800 real GitHub issues and successfully produced accurate summaries most of the time.
Open-source software (OSS)GitHub issuesLarge language models (LLMs)Issue summarizationComment classificationTrajectory synthesisSoftware engineeringAutomated pipelineCollaborative debugging
Authors
Nazia Shehnaz Joynab, Soneya Binta Hossain
Abstract
Resolution of complex post-production issues in large-scale open-source software (OSS) projects requires significant cognitive effort, as developers need to go through long, unstructured and fragmented issue discussion threads before that. In this paper, we present SWE-MIMIC-Bench, an issue trajectory dataset generated from raw GitHub discussions using an automated multi-LLM pipeline. Unlike simple summarization, this pipeline utilizes a group of closed-source LLMs to perform granular tasks: analyzing individual comments with awareness of externally-linked resources, classifying comment analyses into label-specific fields (e.g., root cause, solution plan, implementation progress), and synthesizing label-aware trajectories which capture a structured and coherent narrative of the entire discussion thread. Our pipeline uses five closed-source LLM configurations for distinct purposes: label classification, inline code block and external link summarization, comment analysis, label-specific field classification and trajectory synthesis. By generating concise and reliable trajectories from complex conversation threads, this system can assist developers and researchers of broader software engineering community to understand the experience-driven collaborative approach for issue diagnosis. Furthermore, the generated trajectories can be used to train modern LLM agents to think and act like an expert developer. We evaluated our system on 800 real-world GitHub issues drawn from the SWE-Bench-Pro, SWE-Bench-Multilingual and SWE-Bench-Verified dataset, achieving a 91.7% success rate in extracting 734 high-fidelity reasoning trajectories.