ReferTrack turns “who should I follow?” into a multiple-choice pick over numbered detection boxes instead of an abstract spatial reasoning step, letting a single-camera 4B robot policy match multi-camera baselines on crowded EVT-Bench splits.
Imagine you’re shipping a delivery robot that has to follow one specific person through a crowd based on a text description. The dominant recipe today is a Vision-Language-Action model policy that fuses vision, language, and action into one model, then does chain-of-thought reasoning in some internal latent space before emitting motion. When two people wear similar jackets, that latent reasoning is hard to supervise and often picks the wrong person. The closest prior work, TrackVLA++, reasons through a polar-coordinate token before predicting waypoints, but still keeps identification in an abstract space that isn’t tied to what the detector actually sees.
The core move is to reformulate target identification as a constrained pick from a visible menu. At each timestep the robot runs an off-the-shelf pedestrian detector (YOLO11 plus ByteTrack) on the forward camera, sorts the top-K boxes by area, and hands the model a numbered catalog: <ped_1>, <ped_2>, ..., <ped_K>, <NO_EXIST>. The model’s chain-of-thought is a single token: which index matches the instruction, or <NO_EXIST> if the person isn’t visible. That one token is directly supervised with cross-entropy, unlike a hidden spatial code.
Once a box is picked, its geometry needs to persist so the robot remembers where the target was moving. The authors extend prior Temporal-Viewpoint Indicator tokens tokens (which mark when and from which viewpoint a frame came) by adding the selected bounding box’s coordinates through a small MLP, calling the result TVBI tokens. These get interleaved into the visual history stream. The current frame stays box-free so the model has to re-ground the target each step from raw pixels plus historical motion cues.
To strengthen the picking skill beyond scarce navigation trajectories, they co-train on Refer-QA: static images with 2\u20133 pasted pedestrian crops, each described by a caption, using the exact same indexed-catalog interface. The pipeline per step:
boxes = detector(current_frame) # YOLO + ByteTrack
catalog = index_top_k(boxes) + [NO_EXIST]
hist_tokens = interleave(visual_history, TVBI(prev_boxes))
refer_tok = LLM(instruction, catalog, hist_tokens) # one token
if refer_tok != NO_EXIST:
bbox_queue.push(catalog[refer_tok])
waypoints = ActionHead(LLM(..., refer_tok)) # 2nd pass
During training they occasionally inject a wrong past box into the queue so the model learns to survive its own identification mistakes at inference.
The prevailing fix for language-guided tracking is to let the Vision-Language-Action model model reason in a learned spatial latent, then trust it to emit correct actions. This paper shows the opposite. Make identification a discrete pick from what the detector actually sees, and the reasoning step becomes a single supervisable token that a small model can nail. The cleanest evidence is the ablation on the Distracted Tracking split isolating the referring step from the memory tokens, not the headline benchmark lift.
The load-bearing result is the ablation on Distracted Tracking, where multiple people confuse the tracker. Removing both the Refer-CoT token and the TVBI memory drops success rate to 55.7%. Adding Refer-CoT alone brings it to 70.4%. Full ReferTrack hits 73.3%. So explicit image-space picking is where most of the gain lives; the box-memory tokens add a smaller stabilizing effect. An oracle variant fed ground-truth boxes reaches 81.5%, close to the expert policy’s 85.1%, meaning identification is the bottleneck in crowded scenes, not motion planning.
Secondary numbers on EVT-Bench single-view: 89.4% / 73.3% / 74.1% success on single-target, distracted, and ambiguity splits. Against the strongest prior single-view baseline TrackVLA++, that’s +22.9 points on the hardest split (ambiguity). On identification-heavy splits, the 4B single-camera ReferTrack matches or beats reported three-and-four-camera systems. Real-world rollouts on a Unitree Go2 quadruped and G1 humanoid run at 10.6 Hz end-to-end with detection at 12 ms.
Reach for this pattern when you’re building any embodied agent that has to lock onto one instance among visually similar candidates, based on a text description: a warehouse robot following a specific worker, a service robot tracking a named customer, a drone tailing one athlete in a group. Today you probably wire up a detector, a re-identification model, and a separate planner, and pay the cost of errors compounding across the seams. The pattern here says: make the VLM pick from the detector’s numbered outputs as one token, then plan conditioned on that pick. The identification signal is supervisable, the choice is auditable, and you can pre-train the picking skill on cheap static image data.
Code is at GitHub. The training data comes from expert trajectories curated in Habitat 3.0 using a custom oracle controller (the paper releases the recipe, not the original TrackVLA curation pipeline), plus Refer-QA synthesized from the SYNTH-PEDES person dataset. Backbone is Qwen3-4B with SigLIP and DINOv2 as vision encoders.
When a VLM has to pick one thing among many, give it a numbered menu, not a latent space. Discrete choices over the detector’s own outputs are easier to supervise, cheaper to pre-train on static data, and small models can learn them without reinforcement learning.
•
The whole scheme leans on the pedestrian detector. If YOLO misses the target entirely, no menu entry can be right, and the paper doesn’t stress-test detector failure modes.
•
Gains are largest on identification-heavy splits with multiple similar people. On plain single-target following the win over prior VLA models is real but modest, so the mechanism’s value depends on your deployment actually having ambiguity.
•
Evaluation is almost entirely in the Habitat 3.0 simulator on one benchmark family, with real-world results shown only as qualitative rollouts on two Unitree robots. No quantitative real-world success rates, no comparison against baselines on hardware.