Task Planning for Mobile Manipulation in Retail Stores using Foundation Models with Iterative Re-planning
2026-07-10 • Robotics
Robotics
AI summaryⓘ
The authors explore how robots can help restock shelves in stores by combining advanced language and vision AI models with a specially designed mobile robot. They focus on making the robot understand tasks through user instructions and improve its actions by learning from mistakes. Their approach is tested in a computer simulation where the robot practices picking up and placing items accurately. This moves automation beyond simple, predictable tasks to more complex, real-world retail environments.
automationrobotic manipulationLarge Language ModelsVision-Language Modelsmobile robotstask planningrestockingPyBullet simulationpick-and-place
Authors
Vismay Vakharia, Sanjana Garai, Rolif Lima, Nijil George, Vighnesh Vatsal, Kaushik Das
Abstract
Automation in industries such as retail, warehousing and logistics presents opportunities for greater throughput, cost reduction and mitigation of disruptions from labour shortages. Previously, such efforts have focused on back-room operations involving packing and sorting in relatively structured environments. With advances in robotic mobile manipulation hardware and foundation models, automation can now be applied to more variable and human-centric environments such as retail store shelves. In this work, we present a task-planning approach using Large Language Models (LLMs) and Vision-Language Models (VLMs) to address the restocking problem in retail scenarios such as supermarkets. We demonstrate this system on a custom omnidirectional mobile manipulation platform, with user-driven prompts and a feedback-based iterative re-planning approach for error correction. The end-to-end system is validated in a PyBullet simulation environment for pick-and-place tasks.