Seeing Before Synthesizing: VLM-Guided Transition Event Discovery for Weakly-Supervised Dense Video Captioning
2026-09-03 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionArtificial Intelligence
AI summaryⓘ
The authors study how to automatically find and describe events in long videos using only a list of event descriptions without exact timings. They improve previous methods by creating captions for parts between events only when needed, using a vision-language model to understand what happens in these gaps. Their approach adjusts event boundaries based on where the video's content changes most, making the descriptions better match the visuals. Tests show their method works well on two popular video captioning datasets.
Weakly-Supervised LearningDense Video CaptioningVision-Language ModelsEvent LocalizationTemporal MaskingSemantic VariationActivityNet CaptionsYouCook2 DatasetLarge Language ModelsVision-Language Alignment
Authors
Ye-Chan Kim, Seunghee Choi, SeungJu Cha, Si-Woo Kim, Hwiseon Kim, Hyungee Kim, Dong-Jin Kim
Abstract
Weakly-Supervised Dense Video Captioning aims to localize and describe multiple events in untrimmed videos given only an ordered set of event-level captions per video. Recent work synthesizes auxiliary transition captions via LLM to provide additional vision-language alignment, but these captions lack visual grounding and are rigidly assigned to every inter-event gap at a fixed location and duration. To address these, we propose Seeing Before Synthesizing (SBS), a framework that adaptively provides visually grounded linguistic guidance only where warranted. Leveraging a VLM, we generate frame-level narratives for the inter-event gaps and detect transitions from the semantic variation across them. For identified transitions, we then refine inter-event temporal masks by blending the temporal midpoint with the semantic change point and selecting the width that maximizes vision-language alignment. Experiments on ActivityNet Captions and YouCook2 demonstrate state-of-the-art performance in both captioning and localization.