The Polyglot's Dilemma: Conformance Testing a Dozen Specs in as Many Languages
2026-08-18 • Software Engineering
Software Engineering
AI summaryⓘ
The authors explain how MongoDB's client libraries, written in many programming languages, were hard to keep consistent because each had its own test setup. Over eleven years, they created a system where tests are written once in a common format (YAML) and then run in each language separately. This approach simplified testing by removing a lot of duplicate code and helped reduce bugs in the libraries. They also share insights about designing universal tests and the challenges they faced.
MongoDBclient librariesYAMLspecification-based testingtest automationlanguage interpreterssoftware testingcode consistencytest architecture
Authors
A. Jesse Jiryu Davis, Jeremy Mikola, Jeff Yemin
Abstract
MongoDB maintains client libraries in a dozen programming languages, used by tens of thousands of organizations and millions of developers. Most are implemented natively rather than as wrappers around a shared core. Ensuring consistent behavior across these libraries, comprising millions of lines of code, is hard but essential. Over eleven years, we developed a specification-based testing approach: tests are written once in YAML and executed by language-specific interpreters for each library. We describe the evolution from many ad-hoc formats to a Unified Test Format, which allowed us to delete over 22,000 lines of test code. The rate of nonconformance bugs fell up to 86% in drivers that adopted YAML tests (though results varied). We report lessons learned about declarative test design, test architecture, schema evolution, and the limits of unification.