Event-Based Token Sequences for Audio-Conditioned Music-Game Level Modeling
2026-07-10 • Sound
SoundMultimedia
AI summaryⓘ
The authors look at generating music game levels automatically, which is hard because the levels need to match music timing in a clear way. Instead of breaking music into equal time slices and guessing events for each slice, they represent the levels as sequences of events with explicit timing between them. They use a Transformer model that takes both the music and level info to predict these event sequences, and it performs better than older methods. This approach also helps understand how music cues influence the timing of gameplay events.
Procedural generationMusic game levelsFrame-based representationEvent-based modelingTransformer modelSequence-to-sequenceBeat-shift tokensMultimodal learningRhythm alignmentGameplay events
Authors
Ke Zhang, Chu-Hsuan Hsueh, Kokolo Ikeda
Abstract
Procedural generation of music game levels is an exciting yet challenging problem, as levels must translate musical structure into interactive sequences of timed gameplay events. Most existing approaches formulate this task by frame-based representations, dividing audio into uniform time grids and predicting events at each frame. This makes gameplay events implicit across many frames. As a result, it is hard to describe event-level timing relations and longer-range structure found in human-authored levels. We use procedural generation as a practical setting to study how musical cues map to interactive event sequences. Inspired by event-based symbolic music modeling, we propose a token-level sequence formulation that casts level generation as a multimodal sequence-to-sequence problem. Conditioned on an audio excerpt and level metadata, the model generates a token sequence alternating gameplay-event and beat-shift tokens. This explicitly represents actions and their relative timing in beat space. Based on this formulation, we build a Transformer model. It outperforms representative frame-level baselines under event-level evaluation. It also enables systematic analysis of how audio supports rhythm-aligned event prediction beyond metadata conditioning.