Appearance Pointers -- Multimodal Region Control of Diffusion Transformers
2026-07-21 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionArtificial IntelligenceGraphics
AI summaryⓘ
The authors address the challenge of precisely controlling which parts of an image get certain features during image generation, something hard to do with just text prompts. They introduce 'appearance pointers,' which are small tokens that connect text or images to specific regions the user wants to control. These pointers help guide a type of model called Diffusion Transformers to correctly apply appearances to the right places without needing to retrain the whole system. Their method works with different input types and matches or beats other specialized methods in accuracy.
Controllable image generationDiffusion Transformers (DiTs)Appearance pointersRegion correspondence networkSpatial aggregationMultimodal controlText promptingToken alignmentGenerative image synthesis
Authors
Rahul Sajnani, Yulia Gryaditskaya, Radomír Měch, Srinath Sridhar, Matheus Gadelha
Abstract
Controllable image generation remains challenging for creative professionals, who often require precise regional control over materials, object identities, and spatial arrangements that cannot be reliably achieved through text prompting alone. Diffusion Transformers (DiTs) can natively ingest heterogeneous tokens stemming from texts and images, but they lack mechanisms for determining where and how these tokens should influence the output. We introduce appearance pointers, compact tokens that guide DiTs toward the correct appearance cues at the correct spatial locations by aligning text or image inputs with user-specified masks. Appearance pointers are produced by a region correspondence network and refined through a spatial aggregation mechanism, enabling the model to handle multiple regional descriptions without significantly increasing token load. Our approach introduces the first modality-agnostic interface for localized multimodal control in a DiT without retraining the base model from scratch. Across a range of metrics, our single model reaches or surpasses the performance of modality-specific state of the art methods, offering a simple and extensible path toward precise, region-aware, multimodal guidance in generative image synthesis.