SwanTale is a single flow-matching audio generator that handles both caption-driven (“instruct”) and reference-audio (“zero-shot”) text-to-speech, jointly producing speech, environment, and effects in one waveform via a task-routed Unified MoE and a reward-conditioning trick that fixes the quality signal to “high” at inference.
Suppose you’re shipping a tool that generates dubbed dialogue for short-form video. Today you likely run one model to clone a voice from a reference clip, a second to invent a new voice from a text description, and a third to layer in ambient sound and effects. Timing and reverb drift between the tracks because none of the models know about the others. SwanTale’s pitch is that a single generator can do voice cloning, natural-language voice design, and scene audio in one pass, so the acoustic scene is coherent by construction. The dominant baselines here are CosyVoice 2 style zero-shot cloners on one side and instruct-TTS systems like Parler-TTS on the other. SwanTale merges both jobs into one flow-matching backbone.
Audio is compressed by SwanVAE, a 48 kHz autoencoder that emits 96-dimensional continuous latents at 25 Hz (one latent every 40 ms). A Diffusion Transformer (DiT) denoiser then generates these latents with Flow matching. Captions go through a Qwen text encoder and enter the DiT via cross-attention. For zero-shot, the reference audio’s latent frames are kept fixed and the model fills in the rest; for instruct, the whole latent trajectory is generated from the caption. Two ideas do the heavy lifting. First, reward-conditioned quality control: during training, each sample’s measured audio-quality scores (PESQ, STOI, SI-SDR, MOS) are written into the caption as text and as a discrete flag. The model learns how each quality level sounds. At inference you always ask for “high,” so quality becomes a conditioning knob, not an RL objective. Second, Unified MoE replaces every other feed-forward layer with a mixture of experts routed at two levels: a sample-level task router picks shared experts for instruct vs zero-shot, and a frame-level audio router uses Top-P routing over routed experts plus “null” (skip) experts, so silent frames use less compute than speaker-change frames. Training runs a four-stage curriculum learning path (zero-shot pretraining, dense caption adaptation, full MoE mixture, high-quality SFT), then a Flow-GRPO post-training pass with anchor replay to keep multi-speaker and audio-effect skills alive.
# inference sketch
z_ref = swanvae.encode(reference_audio) if zero_shot else None
latents = gaussian_noise(T, 96)
if z_ref is not None: latents[:len(z_ref)] = z_ref # frozen prompt
caption = append_quality_caption(caption, level="high")
for t in flow_schedule: # solve flow ODE
v_null = dit(latents, t, cond=None)
v_text = dit(latents, t, cond=text_and_turns)
v_full = dit(latents, t, cond=full_caption_or_ref)
v = v_null + w_text(t)*(v_text - v_null) + w_all(t)*(v_full - v_text)
latents = step(latents, v, t)
waveform = swanvae.decode(latents)
The common instruct-TTS design either uses a language-model backbone that swallows both text and captions, or splits speech generation from downstream audio-scene layering. SwanTale takes the opposite bet. One flow-matching denoiser can serve both caption-only and reference-conditioned generation if you separate the routing (task-level shared experts vs frame-level audio experts) and treat audio-quality scores as conditioning rather than as a reward to optimize. The clearest evidence is the SwanBench-Caption ablation isolating Unified MoE, not the headline InstructTTSEval numbers.
On SwanBench-Caption, removing Unified MoE drops Instruction Accuracy from 3.39 to 3.02, Acoustic Quality from 4.31 to 4.09, and Overall Expressiveness from 3.82 to 3.56 (all 1\u20135, judged by a Gemini model). That’s the load-bearing result: it shows the task-plus-audio routing, not just the extra parameters, is what carries instruction-following on hard mixed speech-and-audio prompts. Swapping the caption encoder from an 8B to a 32B Qwen further lifts those scores to 3.70 / 4.34 / 3.98, with the biggest gain on instruction accuracy. Secondary evidence: on InstructTTSEval, SwanTale takes first on Chinese Acoustic-Parameter Specification (86.1) and ties for first on English (84.2), but Role-Play remains weak in both languages. On zero-shot SwanBench-Speech, SwanTale leads Timbre Consistency and both expressiveness metrics in monologue and dialogue, though FishSpeech and SoulX-Podcast still win on content-error and sound-fidelity in their respective settings. SwanVAE reconstruction is competitive across speech, singing, general audio, and music at 25 Hz latents, best-in-class on speech PESQ and MCD.
Reach for this design if you’re building a dubbing or short-video pipeline where a creator types “antique courtyard, faint wind, young woman with a cold tone” and expects speech and ambience to come out time-aligned in one shot, then later wants to reuse that voice via a reference clip. The concrete techniques worth borrowing even outside audio: encoding measured quality metrics as text conditioning and pinning them to “high” at inference, and splitting MoE routing into a sample-level task axis and a frame-level content axis with explicit skip experts for stable regions.
The paper does not link a public code repo, weights, or a released version of SwanData-Caption in the text provided. It also relies on several internal-only assets (SwanData-Speech, SwanVoice, SwanBench-Speech, SwanBench-Caption, SwanVerifier), so external reproduction from the paper alone would be hard. Treat this as an architecture-and-recipe reference, not a drop-in system.
If your generator has to serve two conditioning interfaces (free-text and reference-audio) over heterogeneous content, don’t build two models. Split the mixture-of-experts routing so one axis picks the task and another picks the acoustic regime, and turn your quality metrics into a conditioning input you can pin at test time.
•
Role-Play instruction-following is a clear weak spot in both Chinese and English, and the authors attribute it to gaps in the caption style matrices. If your product depends on “sound like a grizzled detective” style prompts, this recipe as-is won’t get you there.
•
The evaluation leans heavily on LLM-as-judge scoring (Gemini 2.5 Pro, Gemini 3 Pro, gemini-3.5-flash) for expressiveness and instruction accuracy. Absolute score levels and cross-system rankings inherit whatever biases those judges have.
•
Training used roughly 100k hours for SwanVAE and 23M+1.7M hours plus 70M captioned samples for SwanTale on 32\u201364 A100s. The reward-conditioned-quality trick is cheap to copy, but the full recipe is not a small-lab reproduction.