StereoBind generates stereo audio whose perceived left/right location tracks the on-screen motion of the sound source, by conditioning a joint video-audio diffusion model on a 2D motion track through source-binding tokens, absolute track embeddings, and a track-dependent twist on rotary position encoding.
Current joint video-audio generators (think Ovi, LTX-2, MiniMax-H3) decide what sound to play and when to play it, but not where it should appear in the stereo field. If a drone flies left-to-right across the frame, the generated audio typically stays centered or wanders randomly between channels. For VR, AR, and games this breaks the illusion, because humans localize sounds using tiny time and loudness differences between the two ears (Interaural Time Difference (ITD) and Interaural Level Difference (ILD)).
Even models that output two channels do not guarantee meaningful stereo cues: often the left and right waveforms are near-duplicates. The paper calls the missing property dynamic spatial correspondence: the sound’s apparent location should move in lockstep with the visible source. Prior video-to-spatial-audio systems like PrismAudio and See2Sound add stereo as a post-processing step on an already-generated video, without a unified mechanism for binding visual motion to acoustic cues.
StereoBind takes a text prompt plus a 2D motion track (a sequence of (x, y) image coordinates over 121 frames) and generates both the video and matched stereo audio. It builds on the LTX-2 backbone, freezing most weights and adding trainable components plus a small LoRA adapter on the audio branch.
Three mechanisms, each handling a different piece of the spatial problem:
•
Visual Motion Binding (VMB). A set of 64 learnable tokens per transformer block that first attend to the rendered motion reference, then to the current video latent being denoised, then get read by the audio stream. Intuitively, these tokens ask “which visible object is the sound source, and where is it right now?” and hand that answer to the audio branch.
•
Spatial Track Encoder (STE). A small temporal transformer that turns the raw (x, y) track into per-frame embeddings. These embeddings produce a shift, scale, and gate that modulate audio features at each layer, giving absolute position conditioning.
•
Residual Track RoPE (RT-RoPE). The pretrained audio attention uses rotary position encoding keyed on time. StereoBind adds a small, track-dependent phase on top, so the relative rotation between two audio tokens now reflects both their time gap and the spatial displacement of the source between them. A tanh gating gate keeps the perturbation bounded so the pretrained attention does not blow up.
Training uses a new dataset, StereoWorld-29K, built two ways. The VAS pipeline takes real videos, identifies the dominant sound source with Qwen3-Omni, segments it with Grounded SAM, tracks it, recovers 3D geometry with VGGT-Ω, and simulates stereo by placing virtual ears and convolving the mono audio with Room Impulse Response (RIR) computed by gpuRIR. The SDS pipeline uses a Qwen LLM to invent diverse scene specifications and prompts for synthetic video-audio generation.
Pseudo-code for one audio block:
# S: VMB tokens, X_ref: motion-reference feats, X_tar: video latent
S_r = S + cross_attn(S, X_ref) # bind to prescribed track
S_t = S_r + cross_attn(S_r, X_tar) # ground in current video
S = mlp([S_r, S_t, S_t - S_r]) # keep both and their diff
D_s = cross_attn(X_audio, S) # audio reads spatial context
beta, gamma, g = ste(track, layer=ell) # per-layer modulation
X_audio = rmsnorm(X_audio) * (1 + gamma) + beta
X_audio = X_audio + g * D_s # gated residual update
# attention then uses RoPE with added track-dependent phase
On StereoWorldBench, a 95-sample benchmark the authors built with mixed static and dynamic sources, StereoBind improves spatial-fidelity metrics over baselines while keeping video and audio quality comparable. Headline numbers:
•
ILD-W (interaural level-difference Wasserstein distance) drops to 2.754 vs. 3.435 for MiniMax-H3 and 3.744 for Ovi, meaning the loudness-difference distribution between channels is closer to ground truth.
•
SELD-Acc (does a pretrained localizer place the sound in the same horizontal region as the visible source?) hits 0.757 vs. 0.725 for the best baseline.
•
SMR-Err (stereo mid/side energy balance) is 14.86 dB vs. 24.64 for the next-best model, suggesting baselines often generate near-mono audio while labeling it stereo.
•
A human study on 15 clips, 25 raters, five-point immersion scale: StereoBind scores 4.2 vs. 2.3 for LTX-2 and 1.2 for Ovi. Treat this as suggestive given the small sample.
Ablations are informative. Removing VMB causes the largest drop in SELD-Acc (0.506), confirming that binding the audio to a specific visual source matters more than either position-encoding trick alone. Removing STE spikes SMR-Err to 27.38, so absolute position conditioning is what prevents stereo collapse to mono. A separate reversal experiment, where the numerical track and the rendered visual reference disagree, degrades all spatial metrics, suggesting the model actually uses both signals rather than one dominating.
Caveat on the SELD metric: the authors note the localizer was validated mainly on speech and instruments, and about half of StereoWorldBench is speech, so this score is most trustworthy on that slice.
•
If you are building an immersive video generator for VR, AR, or games, this paper argues the right abstraction is a motion track as a first-class conditioning input, not a text hint like “sound moves left to right.” The baseline comparison does exactly that prompt-based workaround and still loses on every spatial metric.
•
If you already have an LTX-2-class backbone, the recipe (VMB tokens plus track-modulated features plus a bounded RoPE phase addition plus a rank-32 audio LoRA) is a cheap retrofit: 8K training iterations on 8 H20 GPUs, about two days. Worth testing if your target use case is horizontal source motion in roughly 5-second clips at 640x384.
•
If you are evaluating stereo audio generation, the paper’s SMR-Err metric is a sanity check worth borrowing: compute the side-to-mid energy ratio and compare to a reference, because models that output duplicated mono can still score fine on audio quality metrics while being spatially useless.
•
For data construction without recording equipment, the VAS pipeline (track a source, recover depth with a monocular 3D model, place virtual ears, simulate RIRs) is a reusable template for turning any mono-audio video dataset into stereo supervision. The project page is at vivocameraresearch.github.io; code release status is not stated in the paper text. •
Evaluation is dominated by horizontal motion in short clips with a single dominant source. The authors list multi-source audio, off-screen sources, camera motion, and reverberant environments as open problems. Vertical motion is sparse in training (combined ~8% of dynamic samples).
•
Spatial supervision in the real-video branch comes from a simulated room (10x10x4 m, T60 = 0.25 s) with the source anchored at a median 1.5 m depth. The model learns the geometry of this simulator, not real recorded binaural audio, so transfer to true recorded stereo is untested here.
•
The headline AST-Ang and AST-Cal metrics reuse a pretrained Spatial-AST model, which has its own biases; treat these as internal-consistency checks rather than ground truth about human spatial perception.
•
Reference numbering in the provided paper text has visible inconsistencies (future-dated arXiv IDs, citation mismatches). The core method and results are self-consistent, but exact prior-work attributions should be verified against the published version.