AI summaryⓘ
The authors created FusionRS, a big dataset that pairs regular RGB satellite images with matching infrared-style images and related captions. These captions include special descriptions of the infrared images to help models understand the unique thermal and lighting features in infrared data. They trained vision-language models on this dataset to better connect images and text across both RGB and infrared views. Their results show that including infrared information and tailored captions improves the model's ability to describe and match these images compared to using only RGB data. This work highlights how adding infrared descriptions helps teach AI to understand remote sensing images in more detail.
Remote SensingVision-Language ModelsRGB ImageryInfrared ImagingCLIPDatasetCaptioningImage-Text AlignmentThermal ImagingModality Fusion
Authors
Jiaju Han, Ben Zhang, Xuemeng Sun, Qike Zhang, Yuxian Dong, Chengyin Hu, Fengyu Zhang, Yiwei Wei, Jiujiang Guo
Abstract
Remote sensing vision-language models have advanced Earth observation understanding, but most existing work remains centered on RGB imagery, leaving the complementary information in infrared data underexplored. Infrared images provide distinctive cues, including thermal intensity structures, object boundaries, and illumination-invariant scene features, which can enrich visual-language learning beyond conventional RGB observations. However, a large-scale RGB-infrared-text dataset for remote sensing vision-language modeling is still absent. To address this gap, we introduce FusionRS, the first large-scale RGB-infrared-text dataset designed for dual-modal vision-language learning in remote sensing. FusionRS is constructed by translating diverse public RGB remote sensing images into infrared-style counterparts, forming aligned RGB-IR image pairs. Each pair is associated with conventional scene captions and IR-aware captions that explicitly describe infrared-specific visual properties while preserving semantic content. Based on FusionRS, we train dual-modal vision-language foundation models for RGB-IR joint understanding. We first train CLIP-style models for RGB-IR-text alignment, and then fine-tune generative VLMs for dual-modal RGB-IR captioning. Experiments show that FusionRS improves RGB-IR alignment, infrared-to-text retrieval, and dual-modal captioning over RGB-only and non-IR-aware training settings. Ablation studies further verify that IR-aware captions are crucial for strengthening infrared-language alignment, highlighting the importance of modality-specific textual supervision for more scalable RGB-infrared remote sensing vision-language representation learning.