StepAudio 3 Music generates full songs up to 5:30 by having a Mixture-of-Experts language model first write a readable ABC notation arrangement plan, then predict a single stream of 50-Hz audio tokens that a flow-matching renderer turns into 48-kHz audio.
Suppose you want an AI to write a complete song, not a 20-second clip. Two things have to happen well at once: long-range musical decisions (verse-chorus form, chord progression, how the melody develops) and moment-to-moment acoustic detail (timbre, vocal texture). Most recent song generators lean on one of two designs. Pure diffusion systems like Stable Audio render long audio directly but give the user only a global text prompt as a steering wheel. Hybrid systems put a language model in front of a diffusion renderer, which helps with structure but still leaves the musical plan implicit inside model activations, so a musician cannot inspect or edit it.
The practical cost: if the chorus lands in the wrong key or the chord progression is boring, your only recourse is to reroll the whole song with a slightly different prompt. StepAudio 3 Music is a bet that exposing the arrangement as human-readable notation, before any audio is synthesized, gives both the model and the user a better handle on the song.
The system has three pieces that were trained separately and then locked together.
First, an audio tokenizer turns 24 kHz audio into one discrete token every 20 ms (50 Hz) drawn from a single codebook of 65,536 entries. This is a deliberate choice against the more common Residual Vector Quantization (RVQ) approach, which stacks multiple codebooks per frame for better reconstruction. The authors ran a controlled comparison: multi-codebook RVQ reconstructs audio more faithfully, but its stacked token streams are harder for a language model to predict, and errors in the first codebook cascade into the others. A single-codebook stream reconstructs slightly worse but is much more predictable, and predictability is what matters when the language model is generating from scratch. The tokenizer is trained in three stages: a self-supervised BEST-RQ warmup, then multi-task supervision (lyrics via CTC (Connectionist Temporal Classification), mel spectrograms for timbre, and chroma for pitch class), then insertion of the quantization bottleneck.
Second, a Mixture-of-Experts autoregressive model predicts those tokens conditioned on lyrics, a text prompt, and optional references. When explicit planning is turned on, the model does two passes: it first emits an ABC-CoT plan (tempo, meter, key, chord sequence, bar structure, melody in ABC notation), appends that plan to its context, then predicts audio tokens conditioned on it. In pseudo-code:
def generate(lyrics, prompt, use_plan=True):
ctx = serialize(lyrics, prompt)
if use_plan:
plan = moe_lm.decode(ctx, mode="abc") # chords, key, melody
plan = optional_llm_edit(plan) # text LLM refines it
ctx = ctx + plan
tokens = moe_lm.decode(ctx, mode="audio") # 50 Hz stream
latents = dit_renderer(tokens) # flow-matching to VAE latents
return vae_decoder(latents) # 48 kHz waveform
Third, a flow-matching Diffusion Transformer (DiT) converts the token stream into continuous latents of the StepAudio VAE, whose frozen decoder produces 48 kHz audio. Long songs are rendered in 30-second chunks with a 2-second latent context carried over for continuity. Interestingly, scaling this renderer from 0.9B to 8B parameters did not help, so the authors kept the small one and put their effort into the tokenizer instead. Post-training uses supervised fine-tuning followed by Direct Preference Optimization on roughly 2,000 pairs of expert preference judgments.
On the authors’ internal objective benchmark, StepAudio 3 Music takes the top score against Suno V5.5, Suno V5, Mureka V9, and MiniMax Music 3 on three AudioBox-Aesthetics axes (Content Enjoyment 7.71, Content Usefulness 8.01, Production Quality 8.39) and on MuQ-MuLan caption-music similarity (0.4465). On SongBench, a seven-dimension musicality benchmark, Mureka V9 wins the individual dimensions in this comparison. So “best perceived quality and best caption adherence” holds; “best on every musical dimension” does not.
On the external, blind Artificial Analysis Music Arena Vocals leaderboard, it lands at Quality Elo 1105, fourth place, behind Suno V5.5 and Mureka but ahead of Suno V5 and the MiniMax models.
Two ablations matter for understanding what does the work. DPO after supervised fine-tuning improves every reported metric (SongBench mean 6.5445 to 6.6438, MuQ-MuLan 0.4157 to 0.4465). And the planning path helps: adding ABC-CoT lifts SongBench mean by +0.0275, and letting a separate text LLM edit the plan before rendering adds another +0.0506, for a total +0.0781 over direct generation. The authors read this as evidence that the plan itself is the bottleneck: the music model is fine at executing a good plan, but its own plans are only okay because it wasn’t trained heavily on music theory. A text LLM with music-theory knowledge can patch that gap.
If you are building a music generation product, the tokenizer finding is the transferable lesson: reconstruction quality of your discrete audio representation is not the right thing to optimize if a language model has to predict it. A single-stream, larger-vocabulary codebook trades a small reconstruction penalty for much more stable long-form generation. Worth testing on your own stack before you commit to an RVQ design.
The ABC-CoT + editing result is the more interesting engineering pattern. You do not need a music-theory-trained generator if you can express the plan in a symbolic form (here, ABC notation) that a general-purpose text LLM can read and revise. This is a cleaner separation of concerns than trying to bake music theory into the audio model itself, and the same shape (symbolic plan, external LLM edits the plan, audio model conditions on the revised plan) could apply to other structured generation domains.
For evaluating competitors, note the authors’ own point: SongBench, AudioBox-Aesthetics, and MuQ-MuLan measure different things and give different rankings. Do not pick one and call it “the” quality metric. The Artificial Analysis arena is the closest thing here to an independent check, and it is a preliminary snapshot rather than a stable leaderboard.
Audio demos are at stepaudiollm.github.io. The paper does not mention released weights or code.
The objective comparison uses one sample per prompt, so sampling variance is not measured. Baselines are queried through hosted product APIs, whose exact inference settings are not reproducible. SongBench was also used to filter the fine-tuning data, so its scores on this model are not fully independent of training. The evaluated subsets are lyrics-conditioned vocal songs; instrumental generation, cover-song synthesis, and vocal-to-mix are supported by the model but not scored comparatively here. Caption similarity and general quality scores do not verify that the audio actually follows the chords, notes, or bars written in the ABC plan; the authors are explicit that note-level adherence is a separate open question. The claim that music-theory knowledge is what the LLM-editing step adds is the authors’ interpretation, not a controlled test.