OmegaUse-OfficeVal: Benchmarking LLM Agents on Long-Horizon Office-Suite Tasks with Economic Grounding
2026-07-29 • Artificial Intelligence
Artificial IntelligenceComputation and LanguageHuman-Computer Interaction
AI summaryⓘ
The authors created a new benchmark called OmegaUse-OfficeVal to test how well large language models (LLMs) can complete complex office tasks that usually take humans a few hours. They included 100 tasks based on real office work and added economic factors like how much time and money humans spend on them. This allows comparing the cost and quality between humans and LLMs. Their tests showed that while LLMs are cheaper and faster, they still don't match the quality of human work. The benchmark, including code and data, is openly available for others to use.
large language modelsoffice-suite workflowsbenchmarkeconomic groundinghuman labor timetask price proxylong-horizon tasksevaluation metricscode-based verificationinference costs
Authors
Jingbo Zhou, Yusai Zhao, Qi Bao, Jingjia Cao, Zhenghai Chen, Chang Gao, Kaiqi Guo, Muxin Guo, Mingxuan Li, Xinjiang Lu, Yanru Ma, Yixiong Xiao, Zenghui Zhang, Le Zhang, Hua Wu
Abstract
Large language model (LLM) agents are increasingly expected to assist users in completing tasks. However, existing benchmarks provide limited support for evaluating whether agents can carry out office-suite workflows at a reasonable cost. We introduce OmegaUse-OfficeVal, a benchmark for evaluating LLM agents on long-horizon office-suite tasks with task-level economic grounding. The benchmark comprises 100 tasks derived from office-suite requests proposed by practitioners and adapted through a privacy-preserving process. On average, these tasks require 2.32 hours of human labor to complete. An important feature of the benchmark is that each task is paired with two economic signals: human labor time and task price proxy. These signals enable direct comparisons between human costs and LLM inference costs, as well as value-weighted evaluation. To support stable evaluation, we develop code-based verifiers from fine-grained rubrics. We evaluate several frontier LLMs together with a human baseline. Although all evaluated LLMs are substantially cheaper and faster than human workers, they have not yet approached human-level deliverable quality. The code and dataset are fully open-sourced, and more information is available on our project website: https://omegause-officeval.github.io.