Motion-Omni makes a talking-avatar LLM emit speech tokens and full-body motion tokens in the same autoregressive pass by conditioning the motion decoder on the Speech Generator’s hidden states instead of the rendered audio, responding 5.4× faster than the equivalent speech-then-motion cascade.
Suppose you’re shipping a voice-driven avatar for a customer-support product. The avatar has to speak and gesture. Today’s default pipeline is a cascade: a spoken-dialogue model like LLaMA-Omni first generates the audio response, then a separate co-speech motion model like EMAGE runs a second full inference pass over that finished waveform to produce the body animation. Two problems fall out: latency roughly doubles, and no gradient from the motion side can ever shape the speech side. This paper asks whether both can be fixed by making motion a first-class output of the dialogue model itself.
The framework has four modules: a speech projector on top of frozen Whisper features, a Qwen2.5-7B-Instruct LLM backbone, a small autoregressive Speech Generator that emits discrete GLM-4-Voice speech units at 12.5 Hz, and a part-aware Motion Generator that emits LOM (Language of Motion) VQ codes at 30 Hz for face, hands, upper body, and lower body.
The key architectural move sits in that Motion Generator. Instead of waiting for audio and re-analyzing it, each per-part decoder attends to the Speech Generator’s last-layer hidden states as keys/values, and uses the embeddings of the just-emitted speech tokens (linearly interpolated from 12.5 Hz up to 30 Hz) as queries. Those hidden states already carry acoustic timing and response semantics, so the model skips the waveform round-trip entirely.
Second move: joint training actually matters. A pilot that froze the Speech Generator and trained only the Motion Generator produced motion visibly out of sync with the audio. Only when the speech pathway is co-adapted with the motion loss does alignment lock in. The authors run a four-stage curriculum (Automatic Speech Recognition, TTS, TTS-with-motion, then a mixed ASR/TTS/speech-to-speech-with-motion/text-to-text stage that unfreezes everything).
Third move: supervision. No captured corpus pairs a single consistent target voice with full-body motion at scale, so they pseudo-label. Run LOM (Language of Motion) as a motion teacher over every response waveform in InstructS2S-200K, score each sample on FGD (Fréchet Gesture Distance)-style Beat Correlation (BC) plus a weighted VQ-VAE reconstruction error, and feed the network progressively larger quality quantiles. Result: 422,856 paired samples, 1,402 hours.
# per training step, joint speech+motion task
h_llm = llm(text_prompt, speech_features) # frozen in early stages
speech_units, h_sg = speech_generator(h_llm) # 12.5 Hz
q = interp(embed(speech_units), 12.5, 30) # up to motion rate
for part in [face, hand, upper, lower]:
motion_codes[part] = decoder[part](q=q, kv=h_sg)
loss = L_lm + L_speech_ce + sum(w[b] * L_motion_ce[b])
The usual way to add gesture to a spoken-dialogue model is a cascade: let the dialogue model finish talking, then run an audio-to-motion model over the waveform. This paper shows the opposite works better. Condition the motion decoder on the internal states that produced the speech, not on the rendered audio, and co-train both losses so the shared representation carries what each head needs. The evidence isn’t the headline benchmark, it’s the pilot ablation where freezing the speech pathway causes motion to visibly drift out of sync, and joint training is what closes it.
The load-bearing result is the frozen-speech pilot: with the Speech Generator held fixed and only the Motion Generator trained, motion loss plateaus above the joint-training level and rendered motion is visibly misaligned with audio. Unfreezing the speech pathway lowers the loss and restores alignment. That’s the finding that makes the thesis true.
Secondary evidence from SwDA-500, their released 500-prompt open-ended dialogue eval set:
•
On latency, Motion-Omni-Q7 completes a full speech+motion response in 4.32 s (RTF (Real-Time Factor) = 0.78, faster than real time). The matched cascade using the same speech but running LOM as a separate audio-to-motion stage takes 23.35 s, i.e. 5.4× slower.
•
On reference-free motion metrics (diversity, beat correlation), the integrated model stays within about 2% of that same-audio teacher cascade despite skipping the teacher at inference.
•
Among systems that do not run LOM at inference, it posts the best beat correlation (7.59) and diversity (13.67) on SwDA-500, plus the best facial and lip-sync scores.
•
Speech quality holds: 2.62% WER on Seed-TTS-Eval, lowest among the omni-modal LLMs compared, and 47.63 on VoiceBench, above LLaMA-Omni, Ex-Omni, and Moshi.
•
Small human A/B study (4 raters, 25 pairs per arm): against MO-audio + EMAGE the pooled margin is +25 wins; against MO-audio + LOM (the teacher itself) it’s roughly a tie.
Reach for this design when you’re shipping a real-time talking avatar and today’s pipeline is “generate audio, then run a separate gesture model over the audio.” The paper’s recipe says: expose the speech decoder’s hidden states as a conditioning signal for the motion head, then co-train under both losses so gradients flow across. You get one inference pass instead of two, and motion timing rides on representations that already know the prosodic plan.
Code and the pseudo-labeled dataset are on GitHub and Hugging Face. SwDA-500 and the evaluation protocol (matched audio across motion systems, shared renderer, automatic + human + latency metrics) are released as the first public benchmark for stochastic open-ended full-body spoken dialogue. The motion teacher is a swappable component, so a stronger teacher can regenerate supervision without changing the framework.
When two modalities have to stay in sync, condition the second one on the first’s hidden states, not on its rendered output, and train them jointly. The waveform is a lossy bottleneck between decoders that already share a plan. Skipping it buys both latency and alignment, but only if gradients from the second loss are allowed to reshape the first module’s representations.
•
Motion quality is capped by the LOM (Language of Motion) VQ-VAE codebook and the teacher’s pseudo-labels. Anything outside that distribution can’t be expressed, and the model can’t exceed its teacher on motion diversity in principle.
•
Training data is English-only and the avatar has one fixed voice with no speaker-identity or emotion conditioning, so cross-lingual and cross-cultural generalization is unverified.
•
The system ingests the full user utterance before responding (offline), so it is not directly comparable to streaming interactive systems, and the human study is small (4 raters, 100 pairs) with the motion-teacher comparison landing near a tie.