AI summaryⓘ
The authors study how to best decide which AI expert model to use for answering a question when checking how good each expert is costs time and resources. They view this as a tradeoff between quickly guessing an expert's usefulness and spending more effort for accuracy, using a known method called Pandora's Box. Their solution, Pandora's Router, smartly chooses when to spend extra effort and when not to, saving work while keeping good performance. They also explore a version where experts decide on their own whether to check themselves first, showing benefits and some drawbacks depending on accuracy. Tests in different AI scenarios show their approach reduces costly checks while maintaining strong results.
Heterogeneous AI systemsRoutingValue estimationPandora's BoxValue of informationGaussian signal modelMulti-LLM benchmarkRetrieval-augmented specialistsInference-time reasoningAllocative efficiency
Authors
Adam Fisch, Shubhendu Trivedi, Fantine Huot, William W. Cohen, Michael Kaisers, Mirella Lapata, Kate Larson, Jacob Eisenstein
Abstract
Heterogeneous AI systems composed of multiple models, architectures, harnesses, or inference-time settings can improve quality and efficiency by routing queries to the specialist who can answer most effectively at the lowest cost. Routing requires estimating each specialist's expected return, but this value estimation has a cost. Cheap estimators (e.g., embedding-based predictors) are fast but noisy, while accurate estimators (e.g., fine-tuned models with access to retrieval results or partial reasoning traces) are expensive. We formalize this tradeoff as an instance of Pandora's Box, the classical problem of optimal search with costly inspection. Under a Gaussian signal model, the resulting policies have closed-form value-of-information expressions that determine, for each specialist and input, whether refining the value estimate is worth its cost. We call the centralized policy Pandora's Router. We extend this to a decentralized setting, Pandora's Bidder, where specialists independently decide whether to invest in self-assessment before accepting an offered price to claim a query. Experiments across three domains---a standard multi-LLM benchmark, retrieval-augmented specialists, and LLMs with variable inference-time reasoning---show that Pandora's Router matches the routing quality of exhaustive estimation, while querying the expensive estimator far less often. In the decentralized setting, value-of-information reasoning improves allocative efficiency when competing estimates are accurate; when competing estimates are noisy, however, it can increase the strategic specialist's utility at the expense of others.