Audio and speech systems: streaming, grounding, generation, and robustnessAudio systems increasingly collapse tasks that once required separate pipelines. A recognizer can emit words and speaker labels together; an audio-language model can connect acoustic evidence to an answer; a generator can coordinate several voices, ambient sound, and video. The central design question is where information must be shared—and when the model must commit to an output.Use this guide by starting with the failure mode you face. For live agents, examine latency, speaker continuity, and accent-sensitive routing. For audio question answering, test whether the waveform actually changes the decision and whether unsupported inputs should be rejected before generation. For content creation, distinguish control over what is produced from control over exactly when events occur.
VibeVoice-ASR-Streaming Technical ReportStart here for a concrete explanation of end-to-end streaming speaker attribution, including chunking, lookahead, retained context, measured label latency, released checkpoints, and comparison with cascaded services.The released checkpoints cover recordings up to eight minutes; uncompressed context grows with session length, and sustained overlapping speech degrades attribution.
What are you trying to solve?
Stream “who said what” without a separate diarizerA live multi-party agent needs both words and stable speaker identity before the conversation moves on. VibeVoice-ASR-Streaming alternates short audio chunks with generated, speaker-labeled text and retains prior audio and text in one autoregressive context. Limited lookahead provides boundary evidence, while conversational history helps reconnect a current turn to an earlier speaker.
VibeVoice-ASR-Streaming Technical ReportStart here for a concrete explanation of end-to-end streaming speaker attribution, including chunking, lookahead, retained context, measured label latency, released checkpoints, and comparison with cascaded services.The released checkpoints cover recordings up to eight minutes; uncompressed context grows with session length, and sustained overlapping speech degrades attribution.
How can one model identify who said what while speech is still arriving?Interleave fixed audio chunks, a small lookahead window, and generated transcript tokens in one context. The model emits a speaker label before the segment text, using acoustic, lexical, and conversational evidence together rather than waiting for a separate clustering stage to revise labels.VibeVoice-ASR-Streaming Technical Report
When is this design a better fit than an ASR-plus-diarization cascade?Consider it when a meeting bot, live captioner, or voice agent needs stable attribution within a few seconds. The published overview reports expected steady-state label latency of 1.53–2.00 seconds for released configurations, but persistent overlap and long sessions remain hard.VibeVoice-ASR-Streaming Technical Report
Verify that audio—not language shortcuts—drives the answerAn audio-language model can produce a plausible answer while barely using the waveform. Two checks address different stages: replace the input with silence or unrelated audio to measure whether predictions depend on acoustic evidence, and place an admission gate before generation to reject waveforms that do not support the requested speech task.
Tracing Audio Grounding and Answer Selection in Audio LLMsRead this to connect counterfactual audio replacement tests with layer-wise changes in answer representations and final predictions. It explains how training can increase acoustic influence rather than merely improve answer accuracy.The abstract supports conclusions for the studied audio answer-selection models and tasks, not open-ended audio behavior.
SURE-Voice: A Front-End Baseline for Speech-Evidence Filtering in Speech LLMsUse this to separate input admission from answer quality. It supplies a benchmark and a simple energy-plus-ASR-confidence gate, while exposing retention and ambiguous-babble trade-offs that a headline rejection rate can hide.Results use LibriSpeech-derived tasks, synthetic unsupported inputs, Common Voice checks, and source-ambiguous babble.
How can you test whether an answer actually depends on the audio?Compare behavior under the real waveform, silence, and unrelated audio. The tracing study reports that acoustic information most affects answer-choice representations in early-to-middle layers, while training increases its influence on final predictions in middle-to-late layers. This helps diagnose where grounding strengthens rather than merely scoring the final answer.Tracing Audio Grounding and Answer Selection in Audio LLMs
Why evaluate unsupported audio before asking the model to answer?Answer-only scoring misses whether the input should have been admitted at all. SURE-Challenge tests a front-end gate on silence, noise, tones, and ambiguous babble. Its energy-plus-Whisper-score rule rejected far more unsupported cases than raw Qwen2-Audio, while threshold tightening reduced retention on Common Voice and babble remained difficult.SURE-Voice: A Front-End Baseline for Speech-Evidence Filtering in Speech LLMs
Control both the content and timing of generated audio scenesCreative control has two separable parts. SwanTale targets what a scene contains—multiple speakers, styles, environmental sound, effects, and fine-grained content—from either instructions or reference audio. Temporal Context Routing addresses when scripted dialogue and shot changes occur by mapping script timestamps onto the shared audio-video generation axis.
SwanTale: Unified Multi-Speaker Speech and Audio Generation for Instruct and Zero-Shot TasksRead this for a unified treatment of instruction-based and reference-based multi-speaker generation, including the roles of captioned data, expressive control, environmental audio, mixture-of-experts modeling, and post-training.The abstract reports generation metrics and demos but gives no licensing, misuse-safety, or production-deployment evidence.
The Missing Temporal Link: Temporal Context Routing for Script-Driven Audio-Video GenerationChoose this when a joint audio-video model stays synchronized but misses scripted event times. It isolates the missing alignment between structured script timestamps and the modalities’ shared temporal coordinates.Timing results come from comparison with a baseline on 200 test scripts.
When should a creator use instruction-driven versus reference-driven generation?Use instruction-driven generation when voices and scene properties must be designed from descriptions. Use zero-shot generation when reference audio should define reusable voice characteristics. SwanTale is useful for understanding a unified model and captioned-data strategy spanning both modes and multi-speaker expressive audio.SwanTale: Unified Multi-Speaker Speech and Audio Generation for Instruct and Zero-Shot Tasks
Why can synchronized audio and video still miss a script’s timeline?Audio and video may share temporal coordinates while script timing remains encoded only as text. Temporal Context Routing projects each scripted event onto that shared axis and routes its guidance to the corresponding positions. On 200 test scripts, the abstract reports large improvements in shot-boundary and dialogue-timing metrics while maintaining baseline synchronization and visual quality.The Missing Temporal Link: Temporal Context Routing for Script-Driven Audio-Video Generation
Prevent accent-sensitive language routing failuresLanguage identification often sits near the front of a spoken-dialogue pipeline, so an accent-driven mistake can route all later processing to the wrong recognizer or language model. Convex Language Detection reframes low-resource adaptation as a convex optimization problem, aiming for sample efficiency, a global optimum, and certified stability to feature perturbations.
Convex Low-resource Accent-Robust Language Detection in Speech RecognitionRead this for a low-resource alternative to standard accent adaptation: a convex language-detection objective with distributed ADMM optimization, sample-efficiency claims, and certified perturbation stability.Reported accuracy concerns language detection under the paper’s low-resource dialect conditions, not end-to-end transcription accuracy.
How can language detection adapt to underrepresented accents without high-dimensional fine-tuning?The CLD paper proposes a convex objective trained with multi-GPU ADMM in JAX. It reports 97–98% accuracy in its low-resource regimes, robustness to dialect variation, and theoretical margin-stability guarantees. Read it when overfitting and scarce dialect data are the main constraints, rather than as evidence about full downstream ASR quality.Convex Low-resource Accent-Robust Language Detection in Speech Recognition