VibeVoice-ASR-Streaming does live “who said what” in a single LLM pass by interleaving 2.9-second speech chunks with generated speaker-labeled text, settling each speaker label in ~2 seconds instead of the 8–51 seconds cloud services take to finalize theirs.
You’re building a voice agent that joins a three-person meeting. It needs to react while people are still talking, and it needs to know who asked the question so it doesn’t reply to the wrong person. The usual stack today is two boxes glued together: a streaming Automatic Speech Recognition engine that emits words, plus a separate Speaker diarization module (often an online EEND variant or a Sortformer-style cache) that clusters voice embeddings to assign speaker IDs. That second box typically revises its labels for tens of seconds after a word was spoken, which is fine for a transcript you read later and useless for an agent that needs to answer now.
Recent LLM-based recognizers like VibeVoice-ASR and SpeakerLM fold both jobs into one generative model, but they read the whole recording first. This paper is the streaming version of that idea.
The model treats a conversation as one long autoregressive sequence that alternates between audio and text: audio chunk 1, its speaker-labeled transcript, audio chunk 2, its transcript, and so on. Everything stays in the LLM’s context, so a speaker who talked five minutes ago is still visible when the model has to decide whether the current voice is that same person.
Each audio chunk is a fixed 2.9 seconds (or 2.0 s in the smaller config), followed by a fixed 0.5-second lookahead so the model can peek just past the chunk boundary before committing text. Audio is turned into latent frames by two frozen encoders from VibeVoice (one acoustic, one semantic, concatenated) at 7.5 frames per second, then projected into a Qwen2.5 backbone at 1.5B or 7B. Speakers are labeled by order of first appearance (Speaker 0, Speaker 1, …) and the label is emitted before the words of that segment, so downstream code knows the speaker from the first token.
Critically, retaining full history isn’t an optimization here. It’s what makes single-model diarization possible. Throw the history away and you have to reintroduce it as an external speaker cache, which reinstates the second stage the design was trying to remove.
context = [] # interleaved audio frames + generated tokens
for chunk in stream_audio(chunk_frames=22):
lookahead = stream_audio(chunk_frames=4) # peek 0.5s
context += [chunk, lookahead, SPEECH_END]
while True:
tok = llm.generate_next(context) # e.g. "\n Speaker 1: hello"
context.append(tok)
if tok == TEXT_CHUNK_END:
break # hand control back to audio
Training goes in three stages using the same objective throughout: first offline speaker-attributed ASR, then streaming pre-training on ~420,000 hours where the sample format switches to the interleaved chunked form, then a ~13,000-hour fine-tune on curated public data plus synthetic multi-speaker meetings (real speakers overlapped and convolved with room impulse responses).
The prevailing intuition from cascaded diarization systems is that speaker labels should be deferred: gather more voice, cluster more embeddings, revise the label once you’re sure. Google’s cloud STT service is the clearest example, revising labels for 16–31 more seconds after first emission. This paper shows the opposite works for an end-to-end LLM. When transcription and attribution share one autoregressive context, the model settles “who is speaking” from lexical and conversational cues as much as from voice, and it can do so inside a single 2.9-second chunk without waiting for the segment to finish. The load-bearing evidence is that emitting the speaker label at the head of a segment is as accurate as emitting it at the tail, which cascaded systems predict should be much worse.
•
Head-vs-tail label placement is a wash: emitting the speaker label before the segment text lands at 31.55 mean cpWER/cpCER against 31.56 for emitting it after. For comparison, Google STT’s own retroactive revision buys it 27.0 to 32.2 cpWER points on the same task, so the effect the end-to-end model shrugs off is genuinely large in the cascaded world.
•
On 12 of 13 speaker-attributed test settings (four meeting corpora plus nine languages of MLC-Challenge), the 7B model gets the best or tied-best cpWER / cpCER, beating Microsoft’s Azure ConversationTranscriber by 2.39 to 12.45 points on the meeting sets while committing labels at an expected 2.00 s versus Azure’s measured 8.21 s and Google’s 9.12 s (first emission) or 51.06 s (final revised).
•
Recognition-only, the five-set mean word/character error is 24.66 versus 25.23 for Gemini 3.5 Transcribe Live and ~39–41 for GPT and ElevenLabs realtime offerings.
•
Streaming costs something compared to the offline VibeVoice-ASR it starts from: WER/CER rises 0.75 to 3.53 points, but cpWER/cpCER rises 5.13 to 6.67, so bounded future context hurts attribution more than transcription.
•
Chunk size and model scale both mainly help attribution, not transcription: going 1.5B → 7B at 22-frame chunks cuts cpWER by 12.76 while cutting WER by only 4.69. Real-time factor stays at or below 0.104 on one A100.
Reach for this if you’re building a meeting bot, a multi-party voice agent, or a live captioner where labels have to be right within a couple of seconds and can’t flicker afterward. Today you’re likely running a streaming recognizer plus a separate online diarizer whose labels churn for half a minute. This model replaces both boxes with one, and the released 7B checkpoint is competitive with Gemini’s live transcription API on raw accuracy while producing stable speaker attribution roughly 4× faster than Azure or Google. It also keeps VibeVoice-ASR’s hotword prompting, so names and jargon can be seeded before a call.
Microsoft has released 1.5B and 7B weights, inference code with vLLM support, and a demo on GitHub and Hugging Face. Two chunk sizes ship (2.9 s at 2.00 s expected latency, 2.0 s at 1.53 s), and the checkpoints support recordings up to eight minutes. Ten languages are covered, limited by what the Qwen3 forced aligner used to build training targets supports.
When one model produces transcript and speaker labels together, waiting longer to be sure who spoke stops paying off. The lexical and conversational context in the LLM does the work that a cascaded system would need extra seconds of voice evidence to do, so you can commit labels at the head of a segment and ship them live.
•
Long simultaneous overlap breaks the design: the decoder serializes speakers onto one output stream, so two people talking over each other for a while degrades sharply. Short conversational overlap is fine.
•
“2.00 s latency” is steady-state. The first label needs one full chunk plus lookahead, so cold-start is 2.5–3.5 s depending on chunk size.
•
Context is kept uncompressed, so compute grows linearly with recording length and the released checkpoints cap at eight minutes. Longer sessions need engineering work the paper doesn’t do.
•
The head-vs-tail equivalence is measured on this training recipe and these benchmarks. It’s plausible that on domains with very short utterances or heavy voice similarity, more voice evidence would still help.