VTaMo: Video-Text Alignment Model for Sign Language Translation
2026-07-10 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionComputation and Language
AI summaryⓘ
The authors developed VTaMo, a system to better translate sign language videos into written text without needing intermediate glosses. Their method improves how the system matches sign video frames to words by aligning them at three levels: fine details between frames and words, overall embedding space adjustments, and better word-level distinctions. They tested this on several sign language datasets and consistently got better results than previous methods. Each part of their approach helps improve translation accuracy.
sign language translationvisual encoderlanguage modeloptimal transportEarth Mover's Distancecontrastive learningembedding spacecross-modal alignment
Authors
Junyi Hu, Zhewen He, Haomian Huang, Aoxiang Yang, Yi Fang
Abstract
Sign language translation (SLT) converts continuous sign videos into spoken language text. Gloss-free approaches leverage pre-trained visual encoders and language models but rely on implicit cross-modal alignment from translation supervision alone. We present VTaMo, a framework that introduces explicit multi-granularity alignment at three levels: (1) local alignment via entropy-regularized optimal transport with a learnable null token for fine-grained frame-to-token correspondences; (2) global alignment via a learnable orthogonal transformation that calibrates embedding space geometry through Earth Mover's Distance; and (3) position-aligned contrastive learning for discriminative token-level representations. Experiments on Phoenix-2014T, CSL-Daily, How2Sign, and OpenASL demonstrate consistent state-of-the-art performance, with ablations confirming the complementary contributions of each component. Code is available at https://github.com/junyi2005/vtamo.