GKDT: General Keypoint Detection Transformer
2026-07-01 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors created a very large and diverse collection of images with detailed keypoint labels, called MegaKPT, by combining many smaller datasets. They built a versatile computer vision model named GKDT using a transformer architecture that can understand images and text prompts to find keypoints on many different objects. They also introduced new training methods to improve the model’s learning. When tested on a wide variety of objects, GKDT performed very well, showing it can work generally across many categories. The dataset and tools will be openly shared for others to use.
keypoint detectiontransformer modelDINOv3visual promptstext promptsmixed-modal trainingdynamic importance samplingPCK@0.1 accuracydataset aggregationcomputer vision
Authors
Changsheng Lu, Yuxin Chen, Haokun Gui, Rong Wang, Jie Yang, Harry Yang, Anton van den Hengel, Jiaya Jia
Abstract
With the emergence of various pre-trained vision and language models, computer vision is shifting from narrow-domain to open-domain recognition. The construction of a more powerful yet general keypoint detection (GKD) model to support diverse tasks has become increasingly important in the field. To this end, we firstly present a large-scale unified keypoint dataset called MegaKPT. The dataset is composed of over 1.3 million diverse object instances from twenty-nine existing datasets, and enjoys high-quality unified annotations with keypoint text descriptions. Based on MegaKPT, we develop GKDT, a simple, flexible and powerful DINOv3 based Transformer model for General Keypoint Detection. Our GKDT supports visual prompts, text prompts, or both. To enhance model training, we also propose a suite of useful strategies such as mix-modal prompted training and dynamic importance sampling. By testing over 22 test sets with seen or unseen objects, our single GKDT model shows strong performance and generality in detecting keypoints on broad categories, with most categories over 90\% PCK@0.1 accuracy, offering high practical applicability to real-world problems. The dataset, models, and codes will be released at https://github.com/AlanLuSun/General-Keypoint-Detection.