LLM4OSC: Profile-Bound Natural Language Control with Deterministic Validation for Open Sound Control
2026-07-28 • Human-Computer Interaction
Human-Computer InteractionSound
AI summaryⓘ
The authors studied how large language models (LLMs) can control audio equipment using Open Sound Control (OSC), a protocol for real-time music and performance control. They found that LLMs often make mistakes like using wrong commands or misreading instructions, which is risky in live shows. To fix this, they created LLM4OSC, a system where the model suggests commands but a strict set of checks and validations ensure only correct messages are sent. Their method achieved perfect accuracy in testing, showing that careful validation is crucial for using language models in sensitive control tasks.
Open Sound Control (OSC)large language modelsreal-time controlparametric controlMax/MSPintent JSONvalidationUDPfew-shot learningpropose-validate-send
Authors
Yuan-Yi Fan
Abstract
Open Sound Control (OSC) is the dominant wire protocol for real-time parametric control in professional audio, live performance, and virtual production. Large language models can emit plausible OSC, but they hallucinate addresses, mishandle type tags, and fail under paraphrase- unacceptable in show-critical contexts. We present LLM4OSC, a local-first architecture in which models propose structured intent JSON over a human-reviewed device profile, and deterministic code validates, clamps, and encodes before any UDP send. We introduce a frozen evaluation harness with CI gates on wrong-send rate: mismatches that would still pass validation and transmit. On a Max/MSP hero profile (12 patterns; 8 literal + 8 paraphrase + 4 refusal cases), after profile tag enrichment, symbolic slot fill, NL refine, and a retrieval confidence gate, backends B0--B3 all pass frozen gates (100% semantic accuracy, 0% wrong-send). B0 (rules) remains the production default at ~0.05ms; LLM backends remain ~3-4s. Historical few-shot B2 accuracy of 62.5% rises to 100% on this suite only after symbolic post-processing- not because the 0.5B model alone becomes show-safe. We argue for propose-validate-send and wrong-send rate as first-class metrics for language-to-control systems.