Video-DeepResearch trains a video agent to actually use its eyes by locking the text-search tools until it has cropped and reverse-image-searched entities across keyframes, lifting visual tool calls per task from 0.10 to 2.33 and letting a 35B open model beat Claude 4.5 Sonnet on video research.
Suppose you ship an agent that answers questions about user-uploaded videos: “who is this person, and what team did they play for in 2019?” The obvious build is to hand the agent a video encoder plus a web-search tool and let it decide what to call. In practice, the paper shows this fails in a specific way. Given the option, VLM agents skip looking at the video and just text-search the transcript or guess from memory. The paper’s baseline number: Qwen3.5-397B-A17B averages 0.10 visual tool calls per task versus 1.27 text calls, and GPT-5 scores 57% on an existing video-research benchmark while making essentially zero tool calls at all. The dominant baseline here is the image-centric Vision-DeepResearch line of work, which added visual tools but never forced the agent to actually invoke them on continuous video.
The core mechanism is a training-time trick: build trajectories where the agent literally cannot see the web-search tool until it has done visual grounding first. The authors call this a decoupled perception-exploration pipeline with stage-wise tool unlocking. Concretely, they synthesize 30K video-grounded QA pairs, then generate 7K successful trajectories where in phase one the action space contains only two tools: Select_Keyframe (pick an informative frame) and Crop_Search (draw a bounding box around an entity and run reverse-image search on that crop). Only after the agent decides it has enough visual context, or hits a perception step budget, does the action space expand to include Search and Visit for the web. Rejection sampling keeps only trajectories that reach the correct answer. To block memorization shortcuts, they discard any QA the base model can answer correctly in 4 tool-free rollouts, guaranteeing the survivors require external tools. Training is two-stage: SFT on the 7K trajectories plus 7K text-only research examples, then Group Relative Policy Optimization (GRPO) on a 2K “moderate difficulty” slice (kept only if pass@4 is strictly between 0 and 1) with a binary correctness reward.
# Trajectory synthesis, per QA pair
history = [query, video_frames]
tools = {select_keyframe, crop_search} # phase 1: vision only
while not agent.says_perception_done(history) and steps < budget:
action = policy(history, tools) # forced to ground entities
history.append((action, execute(action)))
tools |= {search, visit} # phase 2: unlock web
while not agent.emits_answer(history):
action = policy(history, tools)
history.append((action, execute(action)))
keep_if_correct(history) # rejection sampling
The prevailing assumption is that giving a strong VLM the right tools plus scale will produce agentic behavior. The paper shows the opposite. Modality bias is a training-data artifact, not a capability gap: if you never force the agent to use visual tools during trajectory collection, it learns text-only shortcuts even when the answer requires looking at the video. The load-bearing evidence is not the leaderboard row but the tool-usage table: a 30B model trained this way makes more visual calls per task than a 397B baseline.
•
The behavioral shift is the finding that makes the thesis true. On the external VideoDR (Liu et al., 2026) benchmark, their 30B model goes from the baseline’s 0.10 visual calls/task to 2.33, with text calls also rising (1.27 to 4.24). This is a restructured search strategy, not just more calls of the same kind.
•
Ablation on the 30B variant walks the recipe cleanly. Base Qwen3-VL-30B: 40.5% average. Add 7K video trajectory SFT: 53.0%. Add 7K text-only SFT: 56.8%. Add 2K GRPO: 59.3%. Each stage adds a few points; none alone is sufficient.
•
Headline numbers on the paper’s own VideoDR-Bench: their 35B variant hits 64.0% average, versus Claude 4.5 Sonnet at 59.0%, Gemini-2.5-Pro at 57.5%, and GPT-5 at 52.5%. The 30B variant lands at 59.3%, matching Claude.
•
Parametric leakage on the older benchmark was real: on VideoDR (Liu et al., 2026), GPT-5 got 57% with ~0 tool calls. Their newly built VideoDR-Bench, filtered to remove tool-free-answerable questions, forces GPT-5 up to 0.31 visual and 1.43 text calls per task, confirming the benchmark actually requires grounding.
Reach for this pattern when you’re building a multimodal agent that has a strong text-only fallback path. A product example: a shopping assistant that identifies items in user-submitted videos and looks up prices. Your temptation is to give it OCR, visual search, and web search all at once and let the model route. This paper’s evidence says it will mostly route to web search on the caption. The concrete fix is at data-synthesis time: when you generate SFT trajectories, restrict the tool set in phase one to visual tools only, unlock web search only after the model emits a “perception done” token, and rejection-sample on final correctness. You also need a leakage filter: drop any training example the base model can already answer with no tools.
Code and models are on GitHub. The paper releases VideoDR-Bench (a 200-instance multi-hop video QA benchmark, though the paper text alternates between 100 and 200 instances; the exact released split isn’t fully clear) and the two trained models, Video-DeepResearch-30B-A3B and Video-DeepResearch-35B-A3B, both built on Qwen3-VL / Qwen3.5 Mixture of Experts bases. Training details (Megatron, 32 H800s, 80K context) are in the appendix; the SFT + GRPO recipe is fully documented.
If your agent has a text shortcut, it will take it. Remove the shortcut during training, not at inference. Tool-choice policies learned by imitation reflect what the demonstration data made easy, so if you want the model to look at pixels, your trajectory synthesis has to make looking at pixels the only option for a while. Prompting a fully-tooled agent to “please use the visual tools” is not a substitute.
•
The gains are measured against baselines that show near-zero visual tool use, so part of the reported lift is fixing a behavior other systems could also fix with lighter interventions (better system prompts, tool-use SFT). The paper does not run that ablation.
•
The evaluation judge is Qwen3-VL-30B-A3B-Instruct, which is also the base model for one of their trained variants. Same-family judging can inflate scores; a cross-family judge check isn’t reported.
•
The News category regresses on the 35B variant (41.7%, notably below the 30B’s 58.3%), and the authors flag this as unexplained. Whatever the recipe teaches, it doesn’t generalize uniformly across video domains.