AdaBoosting Text Prompts for Vision-Language Models
2026-07-01 • Machine Learning
Machine Learning
AI summaryⓘ
The authors look at how Vision-Language Models (VLMs) use text prompts to classify images, and they notice that current methods don't focus well on examples the model gets wrong. They create a new method called Text Prompt Boosting (TPB), which combines many simple text-based classifiers in a smart way to better handle difficult cases. Their approach improves accuracy on various tasks and works well when applying learned prompts to different VLMs. This means TPB helps make better predictions without needing a lot of extra data or model changes.
Vision-Language Modelstext promptsfew-shot learningAdaBoostensemble learningclassification accuracymisclassified examplescross-model transferprompt engineeringtask adaptation
Authors
Seokhee Jin, Changhwan Sung, Sunung Mun, Hoyoung Kim, Jungseul Ok
Abstract
The classification accuracy of pretrained Vision-Language Models (VLMs) relies on the quality of the text prompts. Handcrafted templates and Large Language Model (LLM)-generated descriptions not only make predictions more interpretable, but also enable reuse of the same prompts across heterogeneous VLMs. Recent works construct task-adapted text prompts with a small number of labeled images. However, existing few-shot text prompting methods do not explicitly focus on misclassified examples during prompt construction, leading to only marginal improvements even as more shots become available. To fully exploit few-shot supervision, we propose Text Prompt Boosting (TPB), an AdaBoost-inspired framework that treats each text-prompt-based classifier as a weak learner and sequentially aggregates them into a strong ensemble by explicitly targeting hard, misclassified examples. Extensive experiments show that TPB preserves task-intrinsic, model-agnostic cues in text space, enabling robust cross-model transfer. Across eleven classification benchmarks, TPB improves accuracy on the source model and preserves shot-driven gains when transferred to larger, more capable VLMs, where existing methods struggle to sustain such improvements.