OpenRFM: Dissecting Relational In-Context Learning
2026-06-03 • Machine Learning
Machine LearningArtificial Intelligence
AI summaryⓘ
The authors study Relational Foundation Models (RFMs), which aim to make predictions on any relational database in a single step. They analyze a known model called the Relational Transformer (RT) and find it struggles when there are few labeled examples, due to its approach to learning relations. By experimenting with training data, they discover that mixing synthetic and real data with better training techniques helps. Based on these insights, they propose OpenRFM, a simpler but more effective model that improves performance by combining different learning stages and smart training strategies.
Relational Foundation ModelsRelational TransformerIn-context learningKernel regressionPre-trainingSynthetic dataFeature learningHomophilyPrototype-based regularizationTabular data
Authors
Zhikai Chen, Junyu Yin, Jialiang Gu, Siheng Xiong, Xiaoze Liu, Ruowang Zhang, Keren Zhou, Kai Guo
Abstract
Relational Foundation Models (RFMs) promise a single pre-trained predictor that, given any relational database, returns predictions in one forward pass via relational in-context learning (ICL). Yet a substantial gap separates open RFMs from their commercial counterparts, and the origin of this gap has not been systematically understood. We dissect a representative framework, the Relational Transformer (RT), from two perspectives. Model side: we show that RT performs relation-level ICL, and a kernel regression view shows it fails when sparse label-cell coverage yields an underdetermined regression. Data side: we ablate RT's pre-training source and find that existing synthetic-only pre-training and in-distribution pre-training drive the same architecture into different regimes, lazy vs. feature-learning. Probing this gap reveals that the missing ingredient is a support-identifiable relational latent in the label-generation process. These two diagnoses translate into (1) a dual-stage ICL architecture that combines the relational backbone with a batch-level ICL layer lifted from a pre-trained tabular foundation model to overcome relation-level label scarcity, and (2) a homophily-aware synthetic plus continual real-data pre-training mixture, augmented with a prototype-based regularization. These choices define OpenRFM, a simple yet effective RFM that improves average task performance by approximately 30% over the RT backbone and surpasses the commercial model KumoRFMv1 on a large set of evaluation tasks.