Video-Oasis audits 14 Video-LLM benchmarks by stripping visual or temporal information and checking if models still answer correctly. 55% of samples turn out solvable without watching the video, and on the remaining video-native samples top models barely clear random-chance.
You’re picking a multimodal model for a product feature that summarizes user-uploaded videos. You compare candidates on published leaderboard scores across Video-MME, LongVideoBench, MVBench, and pick the highest number. This paper argues that number is contaminated: more than half the questions on these leaderboards can be answered from the question text alone, a single frame, an audio transcript, or a bag of shuffled frames. So the score you’re comparing is mostly measuring how well the model exploits linguistic priors and static image cues, not whether it actually understands video. Prior audits like Apollo (redundancy) and Cambrian-S (perception bias) each hit one axis; this work combines visual, temporal, and annotation-quality tests across many benchmarks at once.
The core move is diagnosis by deletion. For each question in a benchmark, run the same question under conditions that should destroy the signal a real video-understanding model needs. If accuracy stays high, the question is a shortcut. There are three families of tests. Visual-dependency tests hand the model only the question (Blind), only a Whisper transcript of the audio, or only concatenated frame captions from CARE. Temporal-dependency tests give only the center frame, randomly shuffle frame order, or use a frozen CLIP-style encoder to top-k match frames to the query without any temporal modeling. Ambiguity tests catch bad annotations: five models disagreeing flags non-unique answers, any 16-frame chunk sufficing flags questions with no real temporal anchor, and humans review cases where shuffled frames still answered correctly (to avoid falsely labeling a legitimately-temporal question as a shortcut). A sample is called a shortcut when a consensus of models (the paper reports thresholds of 1, 2, and all 3) solves it under at least one of these stripped conditions.
for sample in benchmark:
shortcut_votes = 0
for probe in [blind, audio, summary, center_frame, shuffle, bag_of_frames]:
stripped_input = probe(sample.video, sample.question)
if consensus_of_models_correct(stripped_input, sample.answer):
shortcut_votes += 1
if shortcut_votes >= threshold:
mark_as_shortcut(sample)
What remains after filtering (11,033 of 24,416 QA pairs) is then hand-organized into five video-native challenge types: fine-grained perception, spatial world understanding, temporal dynamics and tracking, causal reasoning, and global narrative.
The prevailing habit when a new Video-LLM tops a benchmark is to read the headline number as evidence of stronger video understanding. This paper shows the opposite. A large majority of benchmark questions never required video in the first place, so leaderboard gains mostly reflect better linguistic priors and single-frame recognition, not spatio-temporal reasoning. The load-bearing evidence is not any single benchmark result. It is the correlation between shortcut prevalence and reported accuracy across 14 benchmarks, plus the collapse to near-random when shortcuts are filtered out.
•
Under a relaxed consensus (any one diagnostic model solves the stripped version), an average 92.7% of samples are shortcut-solvable on at least one test. Under strict consensus (all three models), it drops but stays high across task groups.
•
On the filtered video-native subset, most open-source 7B\u20138B Video-LLMs land between 26% and 37% overall, versus a 25.6% random baseline. Gemini-2.5-Pro leads at 46.7%, still moderate.
•
Shortcut-solvable samples aren’t just the easy ones. Overlap between the Video-Oasis-filtered set and a difficulty-baseline (questions three strong models all get wrong) is only 44.6%.
•
An oracle giving ground-truth temporal segments lifts Eagle2.5 from 35.0% to 50.8% on video-native samples but only 78.0% to 80.8% on shortcut samples. Precise temporal grounding matters exactly where the shortcuts have been removed.
•
Adaptive reasoning depth beats always-thinking: an oracle that picks the better of instruction-mode and thinking-mode per question reaches 46.2%, roughly matching Gemini-2.5-Pro, using an 8B base.
•
On training paradigms, long-context Supervised Fine-Tuning (Eagle2.5) beats a plain RL with verifiable rewards recipe (Video-R1), but RL with verifiable rewards with a grounding reward (VideoAuto-R1) helps most on Global Narrative (21.2% \u2192 28.6%). No single training recipe wins across categories.
Reach for this when you’re picking a video model for a product and want an honest ranking. Instead of trusting a raw benchmark score, run the Video-Oasis filters on your eval set to strip questions solvable from transcript, captions, a center frame, or shuffled frames. Then re-rank candidates on the surviving subset. Expect your rankings to move: strong models on the raw leaderboard can collapse toward random when the shortcuts are gone, and adding temporal grounding (frame retrieval like AKS) starts to matter much more once shortcuts are removed.
The pipeline is open-sourced and configurable, so you can plug in new benchmarks or new diagnostic model ensembles (the paper shows ~90% shortcut-set overlap under alternative diagnostic models). The distilled 11,033-sample video-native evaluation set, spanning clips from under 15s to 10min+, is directly usable as a harder eval.
If a benchmark question can be answered without watching the video, it isn’t measuring video understanding, and no leaderboard climb on it is either. The corollary for practitioners: before you trust a video model on a new task, ablate the video. If accuracy barely moves when you swap in a transcript or a single frame, you’ve been benchmarking language, not perception.
•
The diagnostic probes are themselves imperfect. Captions and audio transcripts can smuggle in temporal cues, and shuffled frames sometimes still permit ordering, which is exactly why the authors add human review. Fully automated auditing would leak.
•
“Shortcut” is defined by whether current strong models solve the stripped version. That definition drifts as models improve, so shortcut ratios reported here are a snapshot, not a fixed property of the benchmarks.
•
The 55% headline aggregates across 14 heterogeneous benchmarks; per-benchmark filtering ratios range widely (from ~34% on VSI-Bench to ~77% on EgoSchema). Don’t read it as a uniform tax on every dataset.