TimeLens2 teaches a video model to answer “when did this happen?” across long, multi-moment clips by scoring predictions with a 1-Wasserstein distance-based reward that gives partial credit to near-miss intervals, rescuing 75.8% of training batches where the standard overlap reward would give every candidate a zero.
You’ve shipped a product that lets users ask questions about hour-long recordings: security footage, lecture archives, sports replays, meeting transcripts. The model answers fluently but never cites when in the video it saw the evidence. Users still have to scrub the whole timeline to verify. That’s the video analogue of a chatbot with no citations.
The dominant fix is to fine-tune with reinforcement learning where the reward is Temporal IoU (tIoU) between predicted and true time intervals. This works when the model’s guess already overlaps the truth. It collapses to zero, and provides no learning signal, when the guess is nearby but disjoint, or when the answer is several disconnected clips and the model gets the count wrong.
Two problems get fixed in tandem: the labels and the reward.
For labels, the authors argue that having one annotator watch a long video and mark all evidence intervals in one pass is unreliable. Sparse evidence and visually similar distractors cause missed repeats and sloppy boundaries. So TimeLens2-93K builds labels through a cascade: hierarchical timestamped captions propose candidate intervals; two independent grounding agents (Qwen3-VL-30B and TimeLens-8B) re-localize each proposal from the raw video; only intervals both agents agree on (merged-support IoU > 0.9) survive; a text-video embedding check confirms the query actually matches the clip; then a stronger model refines just the boundaries in a ±3-second window. About 735K raw candidates become 93K verified interval sets.
For the reward, the key move is to represent an interval set as a probability distribution along the timeline: uniform mass over the merged support, zero elsewhere. Two predictions with the same total covered region get the same distribution, even if one is split into three spans and the other into two. The reward is then the exponentiated negative 1-Wasserstein distance distance between the predicted and target distributions, normalized by target duration. In 1D this has a closed form: the integrated absolute difference of the two cumulative distributions. It’s added to the standard tIoU reward, with a penalty for unparseable outputs. Training uses Group Relative Policy Optimization (GRPO) on top of supervised fine-tuning.
def reward(pred_intervals, true_intervals):
if not parseable(pred_intervals):
return -1
p_support = merge(pred_intervals) # union of spans
t_support = merge(true_intervals)
r_iou = len(p_support & t_support) / len(p_support | t_support)
mu_p = uniform_over(p_support) # density = 1/length
mu_t = uniform_over(t_support)
w1 = integrate(abs(cdf(mu_p) - cdf(mu_t)))
r_tw = exp(-w1 / (len(t_support) + eps))
return r_iou + r_tw
The prevailing approach treats reinforcement learning for video grounding as “reward what overlaps the truth” via Temporal IoU (tIoU). This paper shows the opposite. The most useful signal is a graded distance to the target when there is no overlap at all, because that’s where a policy-gradient method has nothing to learn from otherwise. The evidence that carries this thesis is not the leaderboard number; it’s the diagnostic showing that adding the Wasserstein term turns 75.8% of previously-tied training groups into ranked ones.
The load-bearing finding is the group-relative diagnostic on Group Relative Policy Optimization (GRPO). With tIoU alone, 13.8% of rollout groups have identical rewards for every candidate, so mean-centering wipes the gradient. Adding the Wasserstein reward drops that to 3.6% and rescues 75.8% of all-zero-tIoU groups into ranked signal; within-group reward variance grows 4x. Stratifying zero-overlap misses by distance shows the mechanism does what it claims: the reward recovers positive overlap on 21.9% of near misses, 15.8% of mid-distance ones, and only 5.7% of far ones.
•
Against the natural alternative, NGIoU (Normalized Gaussian IoU) with one-to-one interval matching, the Wasserstein reward wins by 0.6 mIoU on average and by 1.4 points on the multi-span VUE-TR benchmark, where matching breaks under fragmentation.
•
On seven benchmarks spanning short, long, multi-span, question-form, and egocentric video, TimeLens2-4B beats every prior open-source model on six of seven, including one with 397B parameters (+7.5 mIoU average).
•
The label-curation cascade adds +3.8 mIoU over raw single-annotator labels while shrinking the training set roughly 8x, with boundary refinement alone contributing +1.7 of that.
•
The declarative-only training data transfers to question-form queries on MomentSeeker, lifting mIoU from 15.3 to 25.8.
Reach for this when you’re building any retrieve-and-cite system over long video: compliance review of surveillance clips, e-discovery over deposition recordings, or an assistant that answers questions about a lecture by pointing to timestamps. Today’s dominant recipe is to fine-tune with tIoU rewards and hope the model’s initial guesses land close enough to bootstrap. TimeLens2 says: give the model a reward that ranks a 30-second miss above a 30-minute one, and represent multi-span answers as merged supports so the reward doesn’t punish equivalent partitions.
Weights (2B, 4B, 8B), the 93K training set, and inference code are released: GitHub, Hugging Face collection, project page. The 2B model already beats every size-matched baseline on all seven benchmarks, so it’s tractable to deploy. The Wasserstein reward is a drop-in replacement for tIoU in any Group Relative Policy Optimization (GRPO) pipeline: it’s a few lines of NumPy because 1D Wasserstein has a closed form.
When your RL reward keeps collapsing to zero, the fix isn’t a better policy, it’s a reward that measures distance to correct, not just presence of correct. Overlap-based rewards work once the model is already close; they can’t teach a cold-start model to get close in the first place. Reward shape matters most in exactly the regime where the base signal is silent.
•
The gains depend on a strong backbone (Qwen3-VL) that already produces parseable interval outputs after supervised fine-tuning. On a weaker base model that mostly emits malformed timestamps, the invalid-output penalty dominates and the geometry-aware reward has less to shape.
•
The label pipeline is only as good as its two grounding agents. The authors note explicitly that stronger proprietary models could improve annotation quality; the corpus is a demonstration, not a ceiling.
•
The train-test overlap audit found zero shared YouTube IDs but couldn’t rule out content duplicates hidden by renamed identifiers, particularly on MomentSeeker. Some fraction of the reported lift on that benchmark could reflect near-duplicate exposure.