Action-Aligned Retrieval with Pairwise Multimodal Reranking for Text-Based Person Anomaly Search
2026-08-24 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors address the challenge of finding people based on specific actions described in text rather than just appearance. They introduce ActPair, a three-step method that improves matching by first training a model to understand actions, then using both the original and a rewritten text query to search, and finally reranking results by directly comparing pairs of images. This approach helps bridge the gap between poses and their descriptions and performs better than existing methods on a public test dataset while also working well on unrelated data. Their method balances accuracy with computing effort efficiently.
text-based person searchaction recognitionvision-language modelmulti-task learninglarge language modellate fusionrerankingpairwise comparisonmultimodal retrievalPedestrian Anomaly Behavior dataset
Authors
Thanh-Khoi Nguyen, Thanh-Nhan Vo, Trong-Thuan Nguyen, Minh-Triet Tran
Abstract
Text-based person anomaly search requires distinguishing individuals based on fine-grained, context-dependent behaviors rather than mere appearance. Existing methods struggle to capture these context-conditioned actions, frequently relying on isolated skeletal geometry, discarding raw query details during reformulation, or utilizing absolute pointwise scoring for multimodal verification. To address these limitations, we propose \textbf{ActPair}, a unified three-stage coarse-to-fine framework that combines action-aligned retrieval with pairwise multimodal reranking to bridge the pose-semantic gap. First, we fine-tune a vision-language model (VLM) with an action-aligned multi-task objective that encourages the representations to encode action-discriminative semantics. Second, we perform parallel late-fusion retrieval using the original query and a large language model (LLM)-generated context-grounded rewrite, retaining complementary details from both semantic views. Finally, we propose an efficient off-the-shelf reranking module that leverages a pivot-promote algorithm to perform direct pairwise visual comparisons, mitigating residual spatial and compositional ambiguities without the prohibitive inference costs of exhaustive evaluation. Extensive experiments demonstrate that our framework achieves the best results among the compared methods on the Pedestrian Anomaly Behavior (PAB) public test and transfers effectively to an unseen, non-anomaly-specific dataset.