TRACE-ROUTER: Task-Consistent and Adaptive Online Routing for Agentic AI
2026-07-24 • Artificial Intelligence
Artificial IntelligenceMachine LearningMultiagent Systems
AI summaryⓘ
The authors identify a problem in current systems that choose which large language model (LLM) to use for each small call without considering the whole task outcome. They propose TRACE-Router, which picks an LLM once for the entire task to better learn from the final result, balancing accuracy and speed. This method avoids guessing task difficulty ahead of time and adapts based on delayed feedback. Their tests show TRACE-Router improves performance compared to existing methods and individual models.
large language modelsroutingcontextual banditstask-level supervisionaccuracy-latency trade-offagentic workflowsdelayed feedbackmodel selectionPareto frontier
Authors
Ritik Raj, Souvik Kundu, Sarbartha Banerjee, Dheemanth Joshi, Ishita Vohra, Tushar Krishna
Abstract
Routing to select large language models (LLMs) with different cost-quality trade-offs has become a fundamental deployment feature of enterprise AI. Existing routers, primarily make independent routing decisions for each LLM call. However, agentic applications execute as long-horizon workflows whose quality is determined only by a delayed, task-level outcome. This mismatch prevents per-call routers from correctly attributing feedback to individual routing decisions. Towards mitigating this, we present TRACE-Router, a task-level routing framework that aligns routing with the unit of supervision. TRACE-Router assigns each task to a model once at admission using a contextual bandit, pins all subsequent LLM calls to the selected backend, and updates its policy using the task's terminal reward, jointly accounting for accuracy and latency. By leveraging delayed task feedback, TRACE-Router learns routing policies that adapt to the workload while avoiding explicit task-complexity estimation. Across three agentic benchmarks, TRACE-Router consistently improves the accuracy-latency trade-off, achieving non-dominated Pareto frontier points. On tau2-Bench, it outperforms latency-matched interpolation between individual models by 7-8 accuracy points, while on Terminal-Bench it achieves 7.1 higher accuracy points than the strongest single model baseline with 36% lower latency.