Model Agreement via Anchoring
2026-02-26 • Machine Learning
Machine LearningArtificial Intelligence
AI summaryⓘ
The authors study how much two machine learning models, trained independently, disagree in their predictions, measured by the average squared difference. They create a general method called anchoring to analyze and limit this disagreement. Using this method, they prove that disagreement can be reduced to zero by adjusting natural factors like the number of models stacked, iterations in boosting, neural network size, or tree depth in regression trees. Their results apply first to simple one-dimensional regression and then extend to more complex cases with multiple outputs and different loss functions.
model disagreementsquared error lossstacked aggregationgradient boostingneural network architecture searchregression treesanchoring techniquestrongly convex lossmulti-dimensional regression
Authors
Eric Eaton, Surbhi Goel, Marcel Hussing, Michael Kearns, Aaron Roth, Sikata Bela Sengupta, Jessica Sorrell
Abstract
Numerous lines of aim to control $\textit{model disagreement}$ -- the extent to which two machine learning models disagree in their predictions. We adopt a simple and standard notion of model disagreement in real-valued prediction problems, namely the expected squared difference in predictions between two models trained on independent samples, without any coordination of the training processes. We would like to be able to drive disagreement to zero with some natural parameter(s) of the training procedure using analyses that can be applied to existing training methodologies. We develop a simple general technique for proving bounds on independent model disagreement based on $\textit{anchoring}$ to the average of two models within the analysis. We then apply this technique to prove disagreement bounds for four commonly used machine learning algorithms: (1) stacked aggregation over an arbitrary model class (where disagreement is driven to 0 with the number of models $k$ being stacked) (2) gradient boosting (where disagreement is driven to 0 with the number of iterations $k$) (3) neural network training with architecture search (where disagreement is driven to 0 with the size $n$ of the architecture being optimized over) and (4) regression tree training over all regression trees of fixed depth (where disagreement is driven to 0 with the depth $d$ of the tree architecture). For clarity, we work out our initial bounds in the setting of one-dimensional regression with squared error loss -- but then show that all of our results generalize to multi-dimensional regression with any strongly convex loss.