SepGen adapts a pretrained joint audio-video generator to emit, in one sampling run, the video, the full soundtrack, and one captioned waveform per sound source, by appending stem channels through a LoRA adapter while protecting the original mix channel from any backward information flow.
If you want to take a generated clip of two people talking in a cafe and rerender it from a different camera angle with proper spatial audio, you need each sound source as its own waveform so a spatial renderer can place it in 3D. Today’s joint audio-video generators (the paper’s backbone is from the LTX-2 family) emit one mixed soundtrack: the two voices and the espresso machine are fused into a single track that can’t be moved or removed.
The usual workaround is to run a LASS model like AudioSep on the finished mix, once per source description. That breaks when two sources belong to the same category. Ask separately for “the man on the left” and “the man on the right” and the separator may return the same voice twice, because each query is processed in isolation with no awareness of its sibling.
The core idea: teach the generator to produce the stems alongside the mix in the same denoising run, so the two stems are jointly aware of each other and of the shared mix they must sum to.
Concretely, SepGen takes a frozen joint audio-video Flow matching generator and extends its audio token sequence with two extra channels, one per stem. The original mix channel is protected: it attends only to itself, and the trainable low-rank adapter is gated off on mix tokens so the backbone’s video and soundtrack come out bit-identical to an unmodified run. Stems can read the mix and each other, but no information flows back into the mix.
Two mechanisms keep stems from collapsing into copies of each other. Block-diagonal caption routing gives each stem cross-attention only to its own caption. Cross-Stem Attention Guidance (Cross-Stem Attention Guidance) treats the sibling’s caption as a negative prompt inside each stem’s text cross-attention, pushing the stem’s content away from what the sibling should contain. All three audio channels share the same rotary time positions so token j of every channel refers to the same instant.
A single checkpoint handles both modes, selected by the noise level of the mix channel at inference:
for step_k in schedule: # sigma goes 1 -> 0
if mode == "separation":
z_mix = observed_mix # clean, every step
else: # generation
if sigma_k > 0.97:
z_mix = z_mix_noisy # read current noisy mix
else:
z_mix = z_mix - sigma_k * v_mix_pred # clean estimate
v_s1, v_s2, v_mix, v_video = model(z_s1, z_s2, z_mix, V, c, q1, q2)
z_s1, z_s2 = solver_step(z_s1, z_s2, v_s1, v_s2, sigma_k)
if mode == "generation":
z_mix, V = solver_step(z_mix, V, v_mix, v_video, sigma_k)
Training is two stages. Stage 1 trains only separation (mix held clean). Stage 2 initializes from that checkpoint and 30% of the time keeps the mix clean, otherwise noises it to at most the stems’ noise level, so the stems see both regimes they will encounter during generation. The training set is 4,843 two-source clips assembled from CelebV-HQ dialogues, URMP instrument duets, model-generated pairs, and cross-video composites from VGGSound and MUSIC-21.
The clearest evidence that joint generation matters is on same-category pairs. The paper reports a J-swap score, which measures how much better a stem matches its own caption than its sibling’s (a stem that just copies the mix scores zero). On the generation benchmark with 134 sound scenes, SepGen’s joint run reaches J-swap 2.07 against 1.26-1.27 for FlowSep and AudioSep cascades fed the same mix and captions. On two-speaker scenes, word error of each stem’s transcript against its own scripted line drops to 0.07, versus 0.81-1.03 for cascades. The cascades degrade because they answer each caption independently and can return the same speaker twice.
The critical ablation: running SepGen with each caption in its own standalone pass (no sibling stem, no joint denoising) collapses J-swap from 1.99 to 0.21 and word error jumps from 0.09 to 1.05. The joint pass, not the architecture per se, is what gives the stems complementary content.
Separation holds up on real recordings too. On SAM-Audio-bench (172 YouTube clips), SepGen beats the strongest SAM Audio checkpoint on Judge and on both swap margins. In a listening study on real recordings, SepGen received 0.64 of the forced-choice judgments against SAM Audio’s 0.20, though SAM Audio was running its best-of-eight reranker while SepGen submitted a single draw.
Two caveats on the measurements. Video content barely affects separation metrics (blank or unrelated frames leave scores unchanged); captions and the mix do the work. And SI-SDR, the standard waveform-level metric, is unfavorable to SepGen because its audio Variational Autoencoder regenerates each stem’s phase from scratch. The paper argues for envelope correlation and log-spectral distance instead, on which SepGen leads.
If you’re building a pipeline that needs addressable sound sources from generated video (spatial audio, selective editing, source-level remixing), the recipe here is more interesting than any single benchmark number. A 327M-parameter LoRA on a frozen 22B backbone, trained on ~5k paired examples on one A6000, is enough to add stem outputs without disturbing the backbone’s video or soundtrack. The protected-channel + adapter-gating pattern (frozen projection for the channel you must not perturb, adapted projection for the new channels) is portable to other “add a modality without regression” problems. Code and checkpoints are released at sepgen.github.io. If you’re currently using a language-queried separator on scenes with multiple same-category sources, worth testing SepGen’s separation mode as a drop-in: it processes all queries jointly in one pass. The pairwise constraint matters most when captions are near-duplicates (two speakers, two guitars).
If you have an existing hosted audio-video generator you don’t control, this method does not apply as described: the mechanism requires training a LoRA inside the audio stream of the backbone, not API-level prompting.
Two things the paper demonstrates but does not deploy. The stems’ video attention concentrates on each source’s on-screen location (Figure 3), suggesting a free placement readout, but the authors flag integrating this into an actual spatial renderer as future work. And although the motivation is 4D rerendering, the paper evaluates stem quality, not end-to-end novel-viewpoint audio.
The layout is hardwired to exactly two sources. Scaling to arbitrary N is left to future work and is not a trivial swap of a hyperparameter: caption routing, attention masks, and the training data are all built around N=2.
Stems inherit the acoustics of the recorded scene, including reverberation and distance cues. A spatial renderer that re-places sources at new positions must undo or account for the original room response.
Same-category pairs with identical captions (two of the same instrument) still produce a split, but the caption cannot direct which stem gets which source, and no permutation of the outputs recovers the reference assignment. The addressing power of SepGen depends on captions being discriminative.
Several baseline comparisons are not perfectly apples-to-apples. SAM Audio’s released pipeline samples eight candidates and reranks; the main tables compare SepGen single draws against SAM Audio single draws, with a separate best-of-eight block showing SepGen keeps its lead. The authors argue single draws are fairer; readers who want the reranked comparison should look at the appendix tables.