Beyond Trial-and-Error: Agentic Optimization for Image-to-Video Adherence
2026-08-12 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionArtificial IntelligenceMultimedia
AI summaryⓘ
The authors present a new system called the "Agentic Self-Improvement" framework to make video generation models more reliable and easier to control. Their method improves video creation by automatically refining text prompts using a language model and checking for meaning accuracy and common errors. Then, it fine-tunes settings like random seeds using a smart search guided by quality scores. Tests showed that videos made with this system were preferred more often than those made without it. This work helps make unpredictable video generation tools more practical for professional use.
Image-to-Video (I2V) modelsprompt optimizationmultimodal Large Language Model (mLLM)Davidsonian Scene Graph (DSG)Common Mistake Questions (CMQ)Bayesian optimizationQuality metricsVideo-Text Adherence (VTA) scorestochastic seedsCFG scales
Authors
Aman Tyagi, Hemanth Boinpally, Jonathan Chen, Douglas Gebert, Steven Hickson
Abstract
Modern black-box Image-to-Video (I2V) models offer powerful capabilities in automated content creation, yet their lack of fine-grained control and reliability presents significant challenges in professional workflows. Their inherent stochasticity causes minor variations in textual prompts or hyperparameters to yield drastically different outputs often necessitating inefficient, brute-force trial-and-error processes. To address these limitations, we introduce the ``Agentic Self-Improvement" framework, which reframes video synthesis into a closed-loop, goal-directed optimization. Our framework systematically navigates the generation parameter space using a novel two-stage approach. In the first stage, an iterative prompt optimization loop uses a multimodal Large Language Model (mLLM) to refine the input prompt. This refinement implements two automated evaluations: Davidsonian Scene Graph (DSG) queries ensure semantic adherence, and Common Mistake Questions (CMQ) for artifact detection. At the second stage, we use Bayesian optimization to efficiently co-optimize stochastic seeds and CFG scales. This search is guided by a suite of quality metrics, including the novel Video-Text Adherence (VTA) score derived from the DSG and CMQ evaluations. Our framework significantly outperforms unguided search methods: in human preference studies, videos generated via our agentic approach were strongly preferred over baseline outputs, achieving win rates up to 69\%. This work provides a practical and extensible methodology for enhancing the predictability and control of state-of-the-art video generation models, moving the field beyond speculative curiosities toward reliable, production-ready tools.