Matching Tasks to Objectives: Fine-Tuning and Prompt-Tuning Strategies for Encoder-Decoder Pre-trained Language Models
2026-06-23 • Artificial Intelligence
Artificial IntelligenceComputation and Language
AI summaryⓘ
The authors studied how different training goals affect language models that generate text and answer questions, especially when dealing with common sense knowledge. They created a system called Match Task to Objective (MTO), which helps pick the best training goal for a given task and prepares data automatically. Their approach, including new fine-tuning templates, greatly improves performance, especially when only a few examples are available. They also applied their ideas to prompt-tuning, making it work better. Overall, their work helps customize language models more effectively for specific tasks.
prompt-based learningencoder-decoder modelspre-training objectivesfine-tuningfew-shot learningcommonsense knowledgeprompt-tuningsoft promptsnatural language processingautomatic task adaptation
Authors
Ahmad Pouramini, Hesham Faili
Abstract
Prompt-based learning has emerged as a dominant paradigm in natural language processing. This study explores the impact of diverse pre-training objectives on the performance of encoder-decoder pre-trained language models across generation and question answering tasks, with a focus on commonsense knowledge retrieval and completion. We highlight the benefits of incorporating multiple objectives during both pre-training and fine-tuning stages. We introduce the Match Task to Objective (MTO) framework and methods for determining the appropriate objective for a given task. This framework offers automated methods to prepare task-related data for adaptation through unsupervised training, based on the identified objective. In the fine-tuning stage, we design novel templates that align with the objectives of the pre-training and adaptation stages. When aligned with task requirements, these strategies can achieve a performance gain of over 120\% compared to conventional methods in few-shot settings. They significantly outperform related works in few-shot settings and exceed the baseline even in full-dataset scenarios. Furthermore, we extend this approach to include prompt-tuning methodologies, providing guidance for more effective soft prompt engineering and optimization. Our strategies significantly enhance prompt-tuning performance as well. These insights hold substantial value, precisely guiding the selection and optimization of models customized for specific tasks. Code is available at https://github.com/puraminy/MTO/