Training Diffusion Language Models for Black-Box Optimization
2026-03-18 • Computational Engineering, Finance, and Science
Computational Engineering, Finance, and Science
AI summaryⓘ
The authors look at a way to improve designs using only old data without trying new ones in real life, which is useful in fields like robotics and materials science. They note that previous methods using language models generate designs step-by-step from left to right, but this misses important two-way connections in the design. To fix this, they adapt another kind of language model called diffusion LLMs that can understand these connections better. They create a special dataset and training process to help these models work well with design data and improve the quality of generated designs. Their approach performs better than past methods on small datasets.
offline black-box optimizationdiffusion language modelsautoregressive modelsbidirectional dependenciesmasked-response predictionreinforcement learningDesign-Benchfine-tuningdomain adaptationrobotics design
Authors
Zipeng Sun, Can Chen, Ye Yuan, Haolun Wu, Jiayao Gu, Christopher Pal, Xue Liu
Abstract
We study offline black-box optimization (BBO), aiming to discover improved designs from an offline dataset of designs and labels, a problem common in robotics, DNA, and materials science with limited labeled samples. While recent work applies autoregressive LLMs to BBO by formatting tasks as natural-language prompts, their left-to-right design generation struggles to capture the strong bidirectional dependencies inherent in design problems. To address this, we propose adapting diffusion LLMs to offline BBO to leverage their bidirectional modeling capabilities. However, a domain gap exists between the natural text pre-training of diffusion LLMs and the heterogeneous signals in BBO (prompts, designs, and labels). To bridge this gap, we construct a unified prompt-response corpus and introduce delimiter tokens to explicitly mark field boundaries for domain adaptation. We further propose a two-stage post-training framework to align the diffusion LLM generation with high-label designs. The first stage performs supervised fine-tuning on the unified dataset via masked-response prediction, and the second stage adopts reinforcement learning with rewards defined by label improvements. Our method achieves state-of-the-art results on Design-Bench small-data settings.