Temporal Context Routing (TCR) makes joint audio-video generators actually respect script timing by adding a per-prompt, duration-normalized bias to text cross-attention, cutting shot-boundary error from 1.11 s to 0.042 s on 200 test scripts.
You’ve shipped a video-generation feature that takes a short screenplay and produces a clip: shot 1 from 0.0-2.3 s, a line of dialogue from 1.7-3.1 s, cut to shot 2, and so on. The video and audio come out nicely lip-synced to each other, but the cut lands a full second late and the dialogue starts before the character is even on screen. That’s the failure mode this paper is about.
Current joint audio-video models like LTX-2.3 and Ovi put video and audio on a shared temporal axis, but the script’s timing (“this shot runs from 2.3 to 5.0 s”) lives only inside the text prompt. The text encoder has no idea those numbers correspond to positions on the generation timeline. So the two modalities stay locked to each other while jointly drifting off the requested schedule. Prior work like Presto and ShotAdapter handles local prompts for video only; nobody had wired script timing into both streams at once.
The core move is simple: stop trying to make the text encoder understand timestamps, and instead inject the timing directly into where text meets video and audio, the Cross-attention layers.
For every token in the script, the authors know which prompt it came from (a shot description, a dialogue line, a global style note) and therefore which time interval [start, end] it governs. At each moment t on the generation timeline, they compute a score that peaks at the center of that interval and falls off smoothly toward the edges, normalized by the interval’s duration so a 5-second shot and a 0.5-second shot get the same relative shape. In plain terms: “this token should have maximum influence in the middle of its assigned window, and much less influence far outside it.”
That score gets added as a bias to the cross-attention logits, alongside the usual semantic query-key dot product. Because it’s additive in log-space, it multiplicatively reweights attention without touching the text, query, or key vectors. Overlapping prompts (a dialogue line that crosses a shot boundary) each get their own bias and stay independently controllable. The whole thing is parameter-free at the operator level and trained via LoRA on top of a frozen 22B backbone.
for token_j in script_tokens:
c = (token_j.start + token_j.end) / 2 # interval center
r = max((token_j.end - token_j.start) / 2, 1e-4)
for t_i in video_times + audio_times:
bias[i, j] = -beta * (t_i - c)**2 / (2 * r**2)
logits = (Q @ K.T) / sqrt(d) + bias + padding_mask
One catch: this needs training clips with accurate shot and dialogue timestamps, which raw scraped video doesn’t have. The authors build a coarse-to-fine pipeline where Gemini first labels each clip under a fixed script schema with rough timing, then PySceneDetect snaps shot boundaries to real visual cuts and WhisperX snaps dialogue to word-level speech alignment. Everything rounds to a 0.1 s grid. This yields ~57k training clips.
The prevailing approach to script timing in generative video is to serialize timestamps into the prompt text and hope the encoder figures it out. This paper shows the opposite. Timing is a coordinate, not a word. Route it directly into cross-attention as a per-prompt bias on the shared audio-video timeline, and leave the text stream alone. The cleanest evidence isn’t the headline benchmark, it’s the Intervals as text ablation: same model, same data, but timestamps left inside the prompt, and shot-boundary error jumps by more than an order of magnitude.
The load-bearing result is the matched-operator ablation on the same LTX-2.3 backbone: when timing stays inside the serialized text prompt, Shot Boundary MAE is 0.601 s and dialogue accuracy is 43.8%. Pull those timestamps out and feed them through TCR’s routing bias, and MAE drops to 0.042 s with dialogue accuracy at 84.1%. Two alternative operators built on the same pipeline (a hard interval mask and a Gaussian-weighted Rotary Position Embedding (RoPE) variant) both land around 0.11 s MAE, so the additive-bias formulation matters, not just the fact that timing is broken out of the text.
•
End-to-end vs open-source baselines: against the strongest baseline (LTX-2.3 itself), TCR cuts Shot Boundary MAE by 96% (1.11 s → 0.042 s), lifts Shot IoU from 0.532 to 0.957, and raises Dialogue Acc@0.5 s from 28.3% to 84.1%. At 24 fps, 0.042 s is roughly one output frame.
•
Quality doesn’t regress: VBench Imaging Quality and Aesthetic Quality shift by less than 2.5% relative across matched operators. SyncNet Sync-C is actually highest for TCR among matched runs.
•
Refinement matters: training on Gemini’s coarse timestamps without the PySceneDetect/WhisperX cleanup pushes MAE to 0.375 s and dialogue accuracy down to 37.6%. The operator alone isn’t enough; the 0.1 s grid supervision is doing real work.
•
Human study: 28 raters across 16 blinded pairwise comparisons prefer TCR on all five axes (shot timing, dialogue timing, script fidelity, AV sync, overall), with 72.3% overall preference vs LTX-2.3 and 83.9% vs JoyAI-Echo.
Reach for this when you’re shipping a text-to-video product that takes structured input, a screenplay, an ad storyboard, a short-drama scene, and today you’re serializing timestamps into the prompt and living with the drift. TCR says: keep the text encoder for semantics, but pass the interval [start, end] for each shot and dialogue line through a separate channel that biases cross-attention on both the video and audio branches. Overlapping prompts (dialogue crossing a cut) work naturally because each prompt gets its own bias. The operator adds no parameters and only touches text cross-attention, so it’s a reasonable LoRA-scale adaptation on an existing joint generator rather than a from-scratch retrain.
On artifacts: the paper doesn’t include a code or model release link in the text provided, so treat the recipe as reproducible from the description rather than something you can clone today. The training corpus is 57,022 clips from two unnamed short-drama collections; the eval set is 200 held-out scripts with 640 shots and 441 dialogue lines. The coarse-to-fine annotation pipeline itself (Gemini schema, PySceneDetect for cuts, WhisperX for word-level dialogue alignment, snap to 0.1 s) is directly reusable if you’re building your own script-timed video dataset.
When your model needs to obey a schedule, encode the schedule as a coordinate, not as words in the prompt. Text encoders are for meaning; timestamps belong wired directly into attention on the same axis the generator already uses. Once you separate those two channels, the model stops paraphrasing your timeline and starts following it.
•
Backbone dependence. All results are on a single 22B joint audio-video generator, LTX-2.3. Whether the additive-bias trick transports to autoregressive video models or to generators with very different cross-attention structure isn’t tested.
•
Requires clean interval supervision. The refinement ablation shows that coarse Gemini timestamps alone give up most of the gain. If your domain lacks reliable shot-cut detection or word-level speech alignment, you can’t cheaply reproduce the training data.
•
Evaluation is short-drama-shaped. 200 scripts drawn from a short-drama distribution, dialogue-centric, portrait 704×1280 at 24 fps. Generalization to long-form, non-dialogue audio events, or non-Latin-script languages isn’t demonstrated, and the paper’s Event stream is narrowed to dialogue only in these experiments.