RCORE fixes a specific failure in video action recognition: models learn to predict verbs from the labeled object rather than from motion. Adding synthesized off-distribution verb-object pairs plus a temporal-reversal penalty cuts the seen/unseen accuracy gap from 17 to 9 points on Sth-com.
You’ve shipped a video-tagging model that classifies short clips as (verb, object) pairs, say for a kitchen-robotics dataset or a content-moderation pipeline. It works great on (pour, cup) and (open, drawer) because those pairs dominate training. Then a user uploads a clip of someone closing a drawer, and your model confidently returns open, drawer. That’s the failure mode this paper diagnoses: the model recognized the drawer, and “open” is the verb most often paired with drawers in training, so it skipped the motion evidence entirely.
The task setting is Zero-Shot Compositional Action Recognition, and the dominant baseline is C2C, which does joint verb-object modeling but, as the authors show, still leans hard on co-occurrence.
The intuition: objects are easy to see in one frame, verbs need multiple frames, so under sparse supervision the model takes the shortcut of predicting verbs from objects. The authors attack both root causes.
First, they synthesize training pairs that shouldn’t exist. For a training clip labeled (fold, paper), they take another clip’s object (say a bottle) and paste it into the high-motion regions of the fold-paper clip using a motion-mask estimator from prior work (FAME). The clip now looks like folding a bottle. They give it a soft label mixing both compositions, so the model sees supervision for verb-object pairs the dataset never contained. This is Co-occurrence Prior Regularization. On top, a margin loss pushes the synthesized pair’s score above the frequently co-occurring hard negatives (e.g., (fold, paper) itself), so common pairs stop dominating.
Second, they force temporal sensitivity. They call this Temporal Order Regularization for Composition. Two penalties: (i) minimize cosine similarity between verb features from the forward clip and the reversed clip (opening should not look like closing), and (ii) when frames are shuffled, penalize confident verb predictions via entropy maximization. If the model can still name the verb from scrambled frames, it’s cheating on static cues.
A third trick makes CPR trainable: instead of softmaxing over all possible verb-object pairs (which treats every unseen pair as a negative and tanks generalization), they expand the label space per mini-batch to include only the synthesized pairs actually present.
for batch in loader:
x_mix, y_soft = inject_object_into_motion(x_i, x_j, mask=fame(x_i))
labels_batch = seen_labels | synthesized_labels(batch)
L_comp = cross_entropy(logits[:, labels_batch], y_soft)
L_cpr = margin_loss(score(y_new), score(frequent_hard_negatives))
L_torc = cos(f_verb, f_verb_reversed) - entropy(pred(f_verb_shuffled))
loss = L_comp + L_cpr + L_torc
The prevailing recipe in compositional zero-shot is to build cleverer joint verb-object heads on top of a strong video encoder and assume better pretraining will handle the shortcut. This paper shows the opposite. The shortcut is baked in by the training distribution and the asymmetric difficulty of verbs vs objects; no amount of pretraining removes it, and you have to regularize against it directly at training time. The cleanest evidence isn’t the benchmark lift, it’s the cosine similarity between forward and reversed verb features: C2C holds it at +0.92 (the model is nearly invariant to time-reversal), while RCORE drives it to -0.79.
The load-bearing finding is the shortcut diagnosis itself. On a controlled biased split where each verb is paired with a single object in training, CLIP gets high object accuracy on unseen pairs but verb accuracy drops below uniform-chance, meaning the model doesn’t just fail at verbs on unseen pairs, it predicts them worse than random. This holds even for a randomly initialized model, so it’s the training distribution, not pretraining.
On the diagnostic metrics False Seen Prediction and False Co-occurrence Prediction, the C2C baseline’s FSP climbs from 53% to 63% during training (misclassified unseen samples increasingly collapse to seen pairs), and the seen-unseen accuracy gap grows in lockstep. RCORE keeps FSP flat around 44%.
On the headline numbers: on Sth-com with InternVideo2, RCORE lifts unseen composition accuracy by +4.5 points over C2C and cuts the seen-unseen gap from 17 to 9 points. On the authors’ new sparser benchmark EK100-com, unseen accuracy jumps by +6.9 to +7.0 points depending on backbone. Ablations show CPR alone mainly helps unseen accuracy at the cost of seen, TORC alone helps verbs, and combining them is complementary.
Reach for this when you’re training a video classifier over a factored label space (verb-object, attribute-object, subject-action) and your training data has a long tail of combinations. The direct transfer: if you have (fold, paper) and (squeeze, bottle) but never (fold, bottle), generate synthetic clips that mix the object into the motion region and train on them with soft labels. Separately, if your labels have temporal structure (open/close, push/pull), add a cosine penalty between features from the forward and reversed clip. Both are backbone-agnostic training-time regularizers.
Code is at GitHub. The paper also releases EK100-com, a repurposing of EPIC-KITCHENS-100 into a ZS-CAR benchmark with a 7.5% label coverage ratio (versus 12.8% for Sth-com), which is directly usable if you’re benchmarking compositional generalization in egocentric video.
When your model has an easy cue and a hard cue that predict the same label, it will take the easy cue every time. Regularize against the shortcut, don’t just scale the encoder.
•
The synthesis trick assumes you have a reliable way to isolate motion regions. FAME works for hand-object interaction video; for scenes without a clear moving foreground (talking heads, static-camera surveillance) the pasted object won’t land in a meaningful place.
•
The temporal-reversal penalty only makes sense for verbs whose meaning flips under time reversal. On the Static split of Sth-com, RCORE’s gains shrink because the regularizer’s assumption doesn’t apply.
•
The authors compare against C2C and one recent method (Jung et al.); they don’t test whether a much larger video-language decoder model (e.g., a video-native chat model) would have the same shortcut behavior, only that encoder-based VLMs up to 1B parameters still do.