DreamX-Creator 1.0 is a 7B open-weight joint audio-video generator that denoises both streams in parallel and couples them through Gated Cross-Modal Attention, a per-token, per-head sigmoid gate that decides how much each modality listens to the other at every layer, paired with an autoregressive one-step 2K refiner that upscales with a single denoising pass per temporal chunk.
You’re building a product that generates short video clips for ads or game cutscenes. Today you run one model for video, then a separate Video-to-audio generation pass to dub in footsteps, ambience, and speech. The audio arrives out of sync on impacts and lips, and you have no way to let the visual model react to what the audio track wanted to say.
Most open joint audio-video systems that fix this are 15B to 33B parameters and often gated behind an API. The comparable open baseline the paper positions against is Ovi at 10B. DreamX-Creator claims to be the smallest openly-released model that does native joint generation AND supports local 2K output.
The generator keeps two separate transformer stacks, one for video latents and one for audio latents, each with its own tokenizer and positional encoding. The first half of the network never lets them talk. The second half inserts two cross-attention paths, video-attends-to-audio (A2V) and audio-attends-to-video (V2A), that share a common time axis via temporal Rotary Position Embedding (RoPE) applied to the cross-modal queries and keys.
The interesting piece is how strongly each path fires. After the cross-modal attention computes its head outputs, a sigmoid gate scores every (token, head) pair using both the target token and that head’s attention output. That scalar in (0,1) multiplies the head output before the residual add. So the model learns per-token when a lip needs to listen to phonemes versus when a background pixel should ignore audio entirely.
During training, each sample is randomly assigned an A2V, V2A, or Joint mode. The mode controls which stream is noisier (the noisier one is the target) and applies a stop-gradient on the cleaner conditioning stream so cross-modal loss can’t corrupt the conditioning backbone. Both streams are trained with Flow matching. Training proceeds in three stages: LoRA adapters on the latter half only, then full-parameter pretraining, then high-quality finetuning.
for batch in data:
mode = sample({A2V, V2A, Joint})
sigma_v, sigma_a = noise_levels_for(mode)
h_v, h_a = independent_stacks(video, audio, sigma_v, sigma_a)
for block in latter_half:
if mode in (A2V, Joint): h_v += gate * xattn(h_v, sg(h_a))
if mode in (V2A, Joint): h_a += gate * xattn(h_a, sg(h_v))
loss = flow_match(h_v) + flow_match(h_a)
After pretraining, a reinforcement-learning stage generates groups of candidates per prompt, scores them along video quality, audio quality, prompt consistency, and sync, and routes each reward to its own stream. The cross-modal (sync) advantage is added to both streams so it optimizes the interaction gates directly. This follows the modality-aware approach from OmniNFT.
For 2K output, they train a bidirectional multi-step teacher on synthetically degraded high-res video, convert it to an autoregressive multi-step refiner via teacher forcing on past chunks, then distill it to a 1-step student using Distribution Matching Distillation (DMD). The student self-rollouts during distillation so it learns to correct its own drift. Audio is passed through untouched.
The usual move for cross-modal fusion is to concatenate tokens or apply symmetric cross-attention everywhere in the latter half. This paper’s bet: the strength of audio-video coupling should be learned per-token, per-head, per-sample, because most video tokens don’t need audio and most audio tokens don’t need video, but a few (lips, impacts) desperately do. The gate is what makes this concrete, and the modality-routed RL rewards are what stop one stream from silently degrading while the other improves. The clearest evidence is that on the Verse-Bench evaluation, the 7B model wins on DeSync (temporal alignment) against models 2-4x its size, even while trailing them on raw audio aesthetics.
On Verse-Bench against similar-scale baselines (NAVA 6.3B, UniAVGen 7.1B, Ovi 10B, DaVinci-MagiHuman 15B), the RL-tuned 7B model posts the best DeSync of 0.1351 (next best is NAVA at 0.2342, so roughly a 40% relative gap on temporal sync) and the best LSE-C lip-sync of 7.84. Video quality (VQ 0.657) is competitive with the larger models. WER on speech content is 0.1112 pre-RL, comparable to Ovi’s 0.1053.
Against larger open systems, the story is honestly mixed and the paper admits it. LTX-2.3 at 22B and MiniMax-H3 at 33B both beat DreamX on audio aesthetics (CE, CU, PQ) and on the ImageBind cross-modal semantic score. MiniMax also wins on lip sync (LSE-C 8.74 vs 7.84). DreamX still holds the DeSync advantage and matches on VQ.
The refiner independently improves MUSIQ from 0.59 to 0.71 and MANIQA from 0.30 to 0.44 over the un-refined 7B output, better than FlashVSR, SeedVR, and LTX-2.5 Refiner on those perceptual metrics, while keeping LSE-C and IB closer to the pre-refinement values than the alternatives.
Blind human preference against Ovi, UniAVGen, NAVA, and DaVinci shows DreamX winning on video quality (61-74% win rates). Against Wan2.7, Kling v3, and MiniMax-H3, the picture flips: DreamX loses on audio quality and AV-alignment, though video quality stays close (roughly 45/15/40 win/tie/lose).
Reach for this when you’re prototyping a product that needs synchronized audio and video from a text prompt plus a starting image, and you can’t send data to a hosted API. The typical scenario: a short-form video tool where a creator supplies a keyframe and describes the scene, and you need lip-synced dialogue plus ambient sound in one pass. You can run inference locally, iterate on the Gated Cross-Modal Attention gates or the reward mix, and then plug in the 2K refiner as an optional quality tier.
The authors say they release the 7B generator weights and the 2K Refiner. The paper does not link a specific GitHub URL in the text provided, and it explicitly notes the release is “open weights” but not necessarily under an OSI-approved license, so check the actual repo for terms before shipping. Training data is a mix of Koala-36M, VGGSound, AudioSet, OpenHumanVid, SpeakerVid-5M, Action-100M, Talker-T2AV, and internal data. The paper acknowledges the RL and refiner stages are presented as designs with “explicit validation requirements, not completed empirical claims,” so treat the reported RL numbers as preliminary.
Cross-modal fusion works better as a learned per-token dial than as a fixed architectural coupling. The gate is cheap, and once you have it, the harder problem shifts to how you route reward signals so that improving one modality doesn’t quietly break the other. Modality-decomposed advantages are the practical answer.
•
The 7B model still trails 22B-33B open competitors on audio aesthetics and cross-modal semantics. If your product judges quality primarily on how good the audio sounds in isolation, the size advantage may not pay off yet.
•
The user study explicitly under-samples highly dynamic and compositionally complex scenes, so the win rates against industrial systems reflect easier cases. Don’t extrapolate to hard prompts.
•
The paper’s own conclusion flags that the RL and refiner stages are “stated as designs with explicit validation requirements, not as completed empirical claims.” The RL numbers in the tables should be read as preliminary, not as ablation-grade evidence that modality-routed rewards are what caused the DeSync improvement.