LLM Post-Training as Brownfield Maintenance: An Industrial Perspective on Dataware Engineering
2026-08-31 • Software Engineering
Software EngineeringArtificial IntelligenceMachine Learning
AI summaryⓘ
The authors study industrial post-training, where teams improve an already deployed AI model without retraining from scratch, working within fixed limits on computing and data mixtures. They highlight three main challenges in this process: balancing data mixtures, focusing on performance yield, and managing uncertain integration steps. Their case study shows that by improving how teaching signals are converted into training data, they significantly increased successful supervision attempts. The final tuned model patch improved code generation performance on standard benchmarks while maintaining overall system stability.
post-trainingbrownfield regimeteacher distillationdata mixturemodel patchingcode generationbenchmark evaluationperformance yieldintegration uncertaintytraining data curation
Authors
Gopi Krishnan Rajbahadur, Amir M. Ebrahimi, Boyuan Chen, Ahmed E. Hassan
Abstract
Industrial post-training is a brownfield regime. Teams inherit a deployed checkpoint and must land targeted improvements under fixed compute and mixture budgets without regressing the rest. The maintained artifact is increasingly dataware: behavior governed by a curated post-training mixture, updated via bounded mixture patches rather than clean-slate retraining. From an industrial code-generation improvement effort, we offer a maintainer's perspective on why this work is hard in practice, distilling three recurring challenges, zero-sum mixture design, yield as the binding metric, and end-to-end integration under uncertainty, and arguing that progress depends less on one-off recipes than on an engineering discipline for programming dataware. In our case study, interventions that raised the conversion of teacher distillation into usable training data increased accepted supervision by 2.84 times while using the same solution teacher and four solution attempts per candidate problem. In our primary evaluation, the yield-engineered patch improved CodeForces pass@1 by +2.59 points (+3.11 pass@3) and held-out LiveCodeBench v6 pass@1 by +6.11 (+8.05 pass@3), all statistically significant across 16 stochastic evaluations of each benchmark from one fixed checkpoint per condition, with internal AIME and MATH regression suites within tolerance.