Desktop-Delta Bench: Do Computer-Use Models Understand Desktop GUI Transitions?
2026-07-28 • Artificial Intelligence
Artificial IntelligenceComputer Vision and Pattern Recognition
AI summaryⓘ
The authors created a new test called Desktop-Delta Bench (DDB) to better understand how computer agents perform tasks on desktop interfaces step-by-step. Unlike previous tests that only checked if a task was done, DDB focuses on whether the agent can recognize changes caused by its actions, which helps avoid mistakes and recover from failures. They tested different models on tasks involving multiple apps and found that current systems still struggle to order actions correctly and identify precise actions like clicks and drags. This benchmark helps highlight specific weaknesses in desktop agents and supports improvements in their accuracy and reliability.
computer-use agentsgraphical user interface (GUI)benchmarktask verificationtemporal orderingaction recognitiondesktop automationLinux applicationsfailure recoverymodel evaluation
Authors
Abhishek Pillai, Samir Kumar Nayak, Yuan Chen
Abstract
Computer-use agents (CUAs) increasingly act through desktop GUIs to complete long-horizon tasks. Current benchmarks primarily measure end-task success or single-frame grounding. Neither isolates whether a model can reconstruct the causal, task-relevant transition produced by an action- crucial for rejecting stale observations, verifying progress, and recovering from failure. This is difficult because inference, remote input, app rendering, and screenshot capture are asynchronous: the next observation may be delayed, occluded, transient, or unrelated, then misread as progress and carried into subsequent planning. We introduce Desktop-Delta Bench (DDB), an offline step-level benchmark with 2,013 human-verified instances from novel, multi-app Linux trajectories across ~15 applications and 50 task domains. DDB trajectories targets 3 failure dimensions- state verification, source tracking, and context-aware control- through 2 complementary tasks: 463 3-frame temporal-ordering instances, including 105 with a cross-trajectory decoy, and 1,550 before-after pairs labeled from 5 actions + its payload. We evaluate 8 closed and open-source model families across 32 ordering and 16 single-action settings, observing consistent gaps. Ordering remains unsaturated: best non-decoy and decoy exact-match rates are 65.1% and 65.7%. Task context improves decoy identification by 6.9 percentage points but reduces non-decoy exact match by 2.2 points; error analysis reveals systematic copying of the presented A-B-C order. Single-action results show that inferring the action family is harder than locating it: click F1 is 0.96 vs, 0.76 for drag, while recognized drags are generally localized well. DDB, thus, complements end-to-end benchmarks by filling the missing diagnostic layer between GUI grounding and final task success, enabling targeted improvements to desktop CUA verification, reliability, and recovery.