GRM: Utility-Aware Jailbreak Attacks on Audio LLMs via Gradient-Ratio Masking
2026-04-10 • Sound
SoundArtificial Intelligence
AI summaryⓘ
The authors study how to trick audio large language models (ALLMs) into behaving badly without ruining their normal functions like transcription and answering questions. They find that messing with only certain parts of the audio frequencies, rather than all of them, can make the attack work well while keeping the model useful. They created a method called GRM that smartly picks which audio bands to change, balancing attack success with preserving the model's normal performance. Tests show GRM outperforms other methods by achieving high attack success without degrading the model’s utility as much.
Audio Large Language ModelsJailbreak AttackFrequency DomainMel BandsPerturbationTranscription QualityQuestion AnsweringSemantic PreservationUniversal Perturbation
Authors
Yunqiang Wang, Hengyuan Na, Di Wu, Miao Hu, Guocong Quan
Abstract
Audio large language models (ALLMs) enable rich speech-text interaction, but they also introduce jailbreak vulnerabilities in the audio modality. Existing audio jailbreak methods mainly optimize jailbreak success while overlooking utility preservation, as reflected in transcription quality and question answering performance. In practice, stronger attacks often come at the cost of degraded utility. To study this trade-off, we revisit existing attacks by varying their perturbation coverage in the frequency domain, from partial-band to full-band, and find that broader frequency coverage does not necessarily improve jailbreak performance, while utility consistently deteriorates. This suggests that concentrating perturbation on a subset of bands can yield a better attack-utility trade-off than indiscriminate full-band coverage. Based on this insight, we propose GRM, a utility-aware frequency-selective jailbreak framework. It ranks Mel bands by their attack contribution relative to utility sensitivity, perturbs only a selected subset of bands, and learns a reusable universal perturbation under a semantic-preservation objective. Experiments on four representative ALLMs show that GRM achieves an average Jailbreak Success Rate (JSR) of 88.46% while providing a better attack-utility trade-off than representative baselines. These results highlight the potential of frequency-selective perturbation for better balancing attack effectiveness and utility preservation in audio jailbreak. Content Warning: This paper includes harmful query examples and unsafe model responses.