Fast & Faithful Function Vectors
2026-06-03 • Computation and Language
Computation and LanguageMachine Learning
AI summaryⓘ
The authors studied how to improve 'function vectors' (FVs), which help large language models understand and follow instructions better. They experimented with different ways to pick which parts of the model to focus on, finding that using a method called Layer-wise Relevance Propagation (LRP) made the models both faster and more accurate. They also found that spreading out the way these vectors guide the model works better than just combining them simply. Their work helps make language models follow instructions more effectively.
function vectorsin-context learninglarge language modelsattention headLayer-wise Relevance Propagationgradient-based attributionsteeringmodel efficiencytask representation
Authors
Minh An Pham, Anton Segeler, Thomas Wiegand, Wojciech Samek, Sebastian Lapuschkin, Patrick Kahardipraja, Reduan Achtibat
Abstract
Function vectors (FVs) are task representations elicited during in-context learning that can be used to steer Large Language Models (LLMs). However, design choices in their formulation remain underexplored. In this work, we study the impact of varying FV definitions for instructions along two degrees of freedom: attention head selection and steering. For head selection, using gradient-based attributions with Layer-wise Relevance Propagation (LRP) substantially improves efficiency as well as accuracy. For FV steering, applying it in a distributed manner yields a higher accuracy compared to simple aggregation. Our code is publicly available.