Prompting-MammAlps: Fine-Grained Text-to-Video Retrieval for Camera-Trap Data
2026-07-10 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionArtificial Intelligence
AI summaryⓘ
The authors created a new test called Prompting-MammAlps to help computers find specific animal videos from large wildlife camera collections using text descriptions. They trained a specialized vision model to spot actions in videos and describe them in detailed text. Then, a language model reads those descriptions and matches them with user queries, making the system easier to understand and less likely to make mistakes. Their method performed better than existing techniques at finding relevant videos based on ecological questions.
Text-to-Video retrievalCamera-trap datasetsVision transformerSpatiotemporal action localizationLarge Language ModelEcological dataVideo-language modelsZero-shot learningInterpretabilityPrompting-MammAlps benchmark
Authors
Valentin Gabeff, Baptiste Maquignaz, Jennifer Shan, Sepideh Mamooler, Gencer Sumbul, Blair Costelloe, Devis Tuia, Alexander Mathis
Abstract
Automatically retrieving videos from large camera-trap datasets remains challenging. Text-to-Video retrieval (TVR) methods based on large video-language models (VLMs) have potential to retrieve events of interest by describing them with simple text queries. However, current methods often lack spatiotemporal understanding and do not generalize well to ecological data. In this work, we introduce Prompting-MammAlps, the first camera-trap TVR benchmark, and propose a fine-grained and interpretable TVR method. Specifically, we trained a vision transformer to perform spatiotemporal action localization, and convert its output to structured text, describing each video. Independently, ethology-inspired queries are processed by a Large-Language Model (LLM) based coding agent to parse the structured text per video and retrieve videos accordingly. We harnessed the LLM to use functions from a custom parsing library to minimize the risk of LLM hallucinations and to improve method interpretability. This retrieval approach applied on the Prompting-MammAlps benchmark achieved a set-based F1-score of 34\% on a test set of 135 ecologically-relevant queries and 775 candidate videos. In comparison the best zero-shot VLM achieved a F1-score of 18\%, while also lacking interpretability. Project page: https://cnai.epfl.ch/prompting-mammalps