Dynamic Infilling Anchors for Format-Constrained Generation in Diffusion Large Language Models
2026-06-03 • Computation and Language
Computation and LanguageArtificial Intelligence
AI summaryⓘ
The authors study diffusion large language models (dLLMs), which can focus on context in both directions and generate text in parallel, helping with tasks where text must follow a strict format like JSON. They identify that fixed anchors, which mark where parts of the text start or end, can be too rigid and cause problems like missing or repeated content. To fix this, they introduce Dynamic Infilling Anchors (DIA), a method that guesses where the text should end before filling in the details, making the output more flexible and accurate. Their tests show DIA improves how well the generated text follows required formats and gives better answers on reasoning tasks without needing extra training.
Diffusion large language modelsBidirectional attentionParallel generationFormat-constrained generationFixed anchorsDynamic Infilling AnchorsIterative infillingZero-shot learningGSM8K datasetMATH dataset
Authors
Boyan Han, Yiwei Wang, Yi Song, Yujun Cai, Chi Zhang
Abstract
Diffusion large language models (dLLMs) offer bidirectional attention and parallel generation, enabling them to exploit global context and naturally support format-constrained tasks like parseable JSON or reasoning templates. While straightforward fixed anchors can enforce such constraints, they often impose rigid spans, leading to truncated reasoning or redundant content. To overcome this, we propose Dynamic Infilling Anchors (DIA), a training-free method that dynamically estimates end-anchor positions to adjust generation length before iterative infilling. This flexible mechanism ensures structural correctness and semantic coherence, avoiding the inefficiencies of fixed-span methods. Experiments on reasoning benchmarks demonstrate that DIA substantially improves format compliance and answer accuracy, achieving significant zero-shot gains on GSM8K and MATH. These results establish DIA as a robust pathway toward reliable, structure-aware generation.