Goku: A Million-Scale Universal Dataset and Benchmark for Instruction-Based Video Editing
2026-06-29 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors created Goku, a huge dataset with 2 million video editing examples that go beyond simple color changes to more complex tasks like moving subjects precisely. They made a smart way to build this data by breaking complicated edits into smaller parts and checking the quality carefully. They also built a new model called Goku-Edit that uses a special language model to understand instructions better and separates tasks to improve editing. To test everything, they made a benchmark called Goku-Bench with 1,000 verified examples and new ways to measure editing performance. Their model did better than other open-source ones in following editing instructions.
Instruction-based video editingMulti-task editingData synthesisMask branchAppearance renderingMultimodal Large Language Model (MLLM)Benchmark datasetInstruction followingVideo editing metricsDecoupled network design
Authors
Sen Liang, Cong Wang, Zhentao Yu, Fengbin Guan, Zhengguang Zhou, Teng Hu, Youliang Zhang, Yuan Zhou, Xin Li, Qinglin Lu, Zhibo Chen
Abstract
Existing instruction-based video editing datasets commonly focus on single-task appearance editing, failing to meet the complex creative demands of real-world scenarios. To bridge this gap, we present Goku, a large-scale dataset featuring 2 million high-quality, instruction-aligned video editing pairs, which is the first to extend task boundaries from basic appearance editing to multi-task and structural manipulations(e.g., precise control of subject movement). To tackle the data synthesis challenges inherent in these complex tasks, we design an efficient data synthesis pipeline that decomposes complex edits into controllable sub-problems and introduce a progressive filtering system for data reliability throughout the whole process. Furthermore, we explore the optimal network structures on Goku, and propose Goku-Edit. To deeply comprehend complex editing instructions, Goku-Edit leverages an MLLM as its text encoder and adopts a decoupled dual-branch design: a dedicated mask branch handles structural control, freeing the main branch for appearance rendering. A comprehensive video editing benchmark, Goku-Bench, is also proposed with 1,000 human-verified test cases and 7 novel editing-specific metrics. Evaluated on Goku-Bench, Goku-Edit obtains up to +8% improvement on other open-source models in terms of instruction following.