Semantic Hardness Is Not Visual Hardness: Sign-Aware Hard Negative Mining for Sign Language Retrieval
2026-07-10 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors studied how to improve searching for sign language videos, especially when signs look very similar but mean different things. They found that the problem isn’t the model itself but how it learns from examples that are hard to tell apart. The authors created a new method called Sign-Aware Hard Negative Mining (SAN) that focuses on signs that look alike, rather than just using text differences. This method helped the system get better at distinguishing tricky signs without losing performance on easier cases.
Sign Language RetrievalHard Negative MiningFine-grained RetrievalVisual ConfusabilityEmbedding SpacePHOENIX-2014T datasetNegative SamplingMachine LearningSign Language ProcessingSemantic Similarity
Authors
Junmyeong Lee, Chan Hur, ChangSu Choi, Sukmin Cho, Fitsum Gaim, Eui Jun Hwang, Hoyun Song, KyungTae Lim
Abstract
Sign Language Retrieval (SLRet) enables efficient access to sign language content but remains fragile in fine-grained scenarios where visually similar signs must be distinguished. We show that this limitation does not stem from model capacity, but from ineffective hard negative supervision. Specifically, we formulate fine-grained retrieval failures as a negative distribution mismatch: semantically distinct yet visually confusable signs are rarely treated as hard negatives, while existing text-based mining strategies fail to capture such visual ambiguity. To address this issue, we propose Sign-Aware Hard Negative Mining (SAN), which constructs hard negatives based on visual confusability in the sign embedding space rather than linguistic similarity. Experiments on PHOENIX-2014T demonstrate that SAN substantially improves fine-grained retrieval performance while preserving coarse-grained accuracy, highlighting the importance of aligning negative supervision with visual ambiguity in sign language retrieval.