Split the Labor: Separating Evidence Interpretation from Decision Aggregation
2026-08-14 • Artificial Intelligence
Artificial IntelligenceComputation and LanguageMachine Learning
AI summaryⓘ
The authors point out that when systems use language models to combine information from multiple sources, they often mix two different tasks: understanding each source and then combining those understandings. They propose separating these tasks using a structured format called a four-field evidence tuple to improve clarity and performance. They identify a problem called "count-scale drift" that happens when adding up evidence weights, and suggest a mathematical fix using calibrated log-likelihood ratios. They test this approach on data over time and show it improves prediction accuracy compared to simpler methods.
language modelevidence tuplecount-scale driftlog-likelihood ratiothresholdingAUPRCsequence encodertree ensembleposterior probabilitydiagnostic panels
Authors
Zhelun Wu
Abstract
Systems that ask a language model to reach a conclusion from many sources usually concatenate them into one prompt. This conflates two operations with different requirements. Interpreting a source rewards capacity and context. Combining interpretations rewards fixed arithmetic, comparability across instances, and the option to return nothing. Once separated, the design problem becomes the interface between them. We propose a four-field evidence tuple (hypothesis, reliability bucket, rationale, provenance) and show that fixing it determines both halves. The separation also reveals a failure mode in how such systems combine, which we call count-scale drift. Thresholding a sum of unnormalized weights is exactly posterior thresholding, but at an operating point that slides with the number of sources consulted. The slide grows with reader reliability. When source reliabilities differ, the vote rule and the posterior order instances differently, and no threshold reconciles them. Pooling calibrated log-likelihood ratios addresses both problems. The fix is arithmetic rather than architectural, and applies to a class of rules beyond language models: score-summing triage engines, diagnostic panels scored by counting positives, and additive multi-signal detectors. We then instantiate the principle twice on one longitudinal corpus, once after outcomes resolve and once before. The same partition helps in both, at different granularities: over reading in the first, over learning capacity in the second. There, a small sequence encoder on an easy auxiliary objective plus a tree ensemble carrying the censored survival loss reaches 0.921 AUPRC against 0.805 for a hand-crafted baseline. We separate what transfers from what must be re-estimated per domain, and state five predictions that would falsify the framework, three negative results, and which comparisons remain confounded.