VideoChat3 is a 4B video-language model that compresses video 16× at the vision-tokenizer stage by folding temporal attention into the ViT and burning fewer tokens on boring frames, cutting LLM visual tokens in half and 2048-frame latency from 44s to 20s versus a same-size baseline.
You’re building a product that watches long videos: a meeting recorder, a driving-log summarizer, a live-stream moderator. Today, most open Video MLLM pipelines sample sparse frames, encode each as an independent image, and dump tens of thousands of visual tokens into the LLM. Cost scales quadratically with sequence length, so a 30-minute clip either blows your context budget or gets sampled so aggressively it misses the action. VideoChat3 attacks this at the tokenizer, before the LLM ever sees anything, and releases the full training stack, unlike Qwen3-VL which ships weights only.
Two ideas, both about spending fewer visual tokens without losing information.
First, the vision encoder itself becomes temporally aware. Instead of encoding each frame independently, group 4 consecutive frames into a chunk, run 3D spatiotemporal attention across space and time inside that chunk, then pool along the time axis. Combined with a 2×2 spatial merge, this yields a 16× compression before tokens hit the LLM. The authors call this I3D-ViT. It’s initialized from a pretrained image ViT (MoonViT) with learned temporal position embeddings for frames 0…T-1, so pretrained spatial knowledge survives.
Second, streaming inputs get an adaptive resolution controller. At each step the model emits one of three state tokens: Silence (nothing interesting, keep watching), Standby (something might be happening, look harder), Response (answer now). The Standby token doubles as a signal to encode the next window at 448² pixels instead of 224², so the model spends visual budget only when it thinks a payoff is coming.
for window in stream:
tokens = i3d_vit(window, pixel_budget=b)
state = llm.predict_state(tokens, history)
if state == "Response":
yield llm.generate_answer()
b = LOW
elif state == "Standby":
b = HIGH # inspect next window closely
else: # Silence
b = LOW
Training is a four-stage curriculum ending in streaming instruction tuning. A key trick there: naively training on all state tokens is dominated by repeated Silence labels, so the model learns to never speak. Training only on transitions lets it cheat by copying the previous state. Their fix, state-transition mask, keeps loss on every state change plus an equal random sample of “stay” positions.
The default in video MLLMs is to inherit an image tokenizer, sample frames sparsely, and let the LLM sort out temporal structure downstream. This paper argues the opposite. Compress the video’s spatiotemporal redundancy inside the vision encoder, before tokens ever reach the LLM, and the quadratic cost of long video collapses into a linear cost on the encoder side. The clearest evidence is the efficiency table at 2048 frames, not the leaderboard numbers.
The load-bearing result is the efficiency curve. At 2048 input frames, VideoChat3 processes half as many visual tokens as Qwen3-VL (100K vs 200K), cutting total latency from 44.4s to 20.4s and FLOPs by over 60%. At 1024 frames, latency drops from 12.3s to 8.1s. The vision encoder gets slower in absolute terms, but the LLM stage shrinks so much that total wall-clock improves.
Secondary evidence that the compression doesn’t destroy quality:
•
On general video benchmarks, VideoChat3-4B matches or beats Qwen3-VL-4B on 18 of 19 directly comparable metrics, with large gains on temporal grounding (TimeLens: +6 to +10 points).
•
Streaming: on OVO-Timing, which measures whether the model answers at the right moment, VideoChat3 hits 35.5 F1 vs 8.1 for Qwen3-VL-4B and 31.0 for a specialized system with an auxiliary 2B module.
•
The adaptive-resolution ablation is the cleanest mechanism check. Fixed-low resolution scores 33.5 F1, fixed-high scores 30.5 (worse, because high-res everywhere floods the model with irrelevant detail), and dynamic scores 35.5 while using only 30% of the all-high pixel budget.
•
The state-transition mask ablation: training loss on all state tokens gives 5.8 F1 (the model just outputs Silence); loss only on transitions gives 20.1 with 97.9% recall but 11.8% precision (spams responses); the balanced mask gives 35.5.
Reach for this when you’re shipping a long-form video assistant, a meeting summarizer, a surveillance QA system, or any product where 1000+ frame contexts are normal and inference cost dominates. The recipe generalizes: if your current pipeline runs a per-frame ViT and dumps tokens into an LLM, inflating the ViT into short spatiotemporal chunks and pooling along time is a drop-in halving of your LLM sequence length. The Silence/Standby/Response pattern is also directly reusable for any streaming agent that needs to decide when to speak.
The release is unusually complete: model weights, training code, all three curated datasets (VideoChat3-Academic2M re-annotated from public academic sources, VideoChat3-LV116K for long videos, VideoChat3-OL617K for streaming), and the synthesis pipelines that built them. This matters because the training data recipe, using Qwen3-VL-235B-A22B as both an annotation generator and a consistency judge that filters unsupported rewrites, is arguably as important as the architecture for reproducing the results.
Compress video redundancy in the tokenizer, not the LLM. Spatiotemporal attention inside the vision encoder is cheap and linear in frames; visual tokens in the LLM context are expensive and quadratic. Move the work upstream and long-video inference stops being the bottleneck.
•
The efficiency win depends on the LLM being the dominant cost. On short clips (256 frames in their table), VideoChat3 is actually slower end-to-end than Qwen3-VL because the heavier encoder isn’t yet amortized. The crossover happens around 512+ frames.
•
The streaming controller is a hand-coded deterministic rule (Standby → high-res next window, everything else → low-res). It’s not learned end-to-end, so failure modes where the model under-triggers Standby will silently degrade perception without any signal to correct itself.
•
Benchmarks are almost entirely their own team’s or close collaborators’ (ODVBench, OVBench, TimeLens, River are all cited to authors overlapping with this paper). The general-video comparisons against Qwen3-VL and Molmo2 are more independent but still use configurations the authors chose.