Urbex grounds natural-language references to objects in city-scale 3D scenes by letting a vision-language agent zoom, query landmarks, and render oblique views instead of ranking thousands of precomputed object proposals, running about 10× faster than the strongest ranking baseline.
Suppose you want a drone or map assistant to find “a white car parked next to the orange car to the left of the West Midlands Police Custody Suite” inside a point cloud covering a square kilometer of city with tens of millions of points. Two existing recipes both struggle. Indoor 3D grounding methods downsample the whole point cloud to a fixed budget (around 50,000 points), which erases the few pixels a small car occupies. City-scale methods like CityAnchor instead precompute every candidate object in the scene and score each one against the query, which couples accuracy to the proposal pipeline and makes inference cost grow with the candidate count. Both treat localization as a one-shot retrieval problem, even though a human would scan the skyline for the named building first, then look for cars nearby.
Urbex reframes grounding as a sequential search inside an interactive environment wrapped around the scene. The agent starts from a global bird’s-eye-view image with a known mapping back to world coordinates, then calls one of three tools and reads the returned image before deciding what to do next.
•
Landmark Query resolves a proper noun from the query (a building, road, facility) using OpenStreetMap data and returns its world position plus projection into a chosen view.
•
Zoom-in crops a selected region from the original high-resolution BEV (whose long side often exceeds 10,000 pixels), recovering detail the global view lost.
•
Oblique Rendering renders a candidate region from a cardinal direction to verify 3D shape, height, and facade.
The policy is a 4B-parameter Qwen3-VL model trained end-to-end with reinforcement learning on CityRefer using Group Relative Policy Optimization (GRPO). Reward combines a smooth terminal signal (IoU with the ground-truth box plus an exponential center-distance term so the agent gets gradient even when overlap is zero) with per-tool shaping: zooms are rewarded for moving closer to the target, oblique renders for overlapping the target region, and malformed calls are penalized. Shaping rewards use ground-truth annotations but never appear in the agent’s observations, so they can’t leak the answer at inference.
history = [query, global_bev]
for t in range(max_turns): # budget of 8
thought, action = policy(obs=history[-1], hist=history)
if action.is_final_box():
return project_to_world(action.box, action.view)
obs = env.run_tool(action) # landmark / zoom / oblique
history.append(obs)
On CityRefer, Urbex reports the best Hit@1 on both the novel-description and novel-object splits (25.8 and 23.9 with ground-truth candidates for post-hoc matching), ahead of CityAnchor at 23.4 and 14.2. CityAnchor still edges Hit@5 on the novel-description split because exhaustive ranking retains more plausible candidates in its top five. Per-query online inference is about 3.2 seconds for Urbex versus 32.1 seconds for CityAnchor, with both numbers excluding offline scene preparation.
Transferring the CityRefer-trained policy zero-shot to a newly constructed STPLS3D-Refer set (built from STPLS3D aerial scans plus VLM-generated expressions) keeps Urbex ahead on every reported metric, including a Hit@1 of 27.2.
Ablations isolate what each piece buys. Removing all tools drops Hit@3 from 36.9 to 10.5. The single biggest tool gain comes from Landmark Query (Hit@3 jumps from 15.5 to 28.2 when it’s added on top of Zoom-in), reinforcing that named anchors drive city-scale search. On the reward side, terminal IoU alone reaches 22.8 Hit@3; adding distance shaping lifts it to 30.6, and the full reward reaches 36.9. A diagnostic also shows that applying the learned Landmark→Zoom→Oblique tool order as a fixed script to a prompted GPT-5.4 baseline cuts its token use roughly in half while matching its Hit@1, so part of the value is the discovered search routine rather than the trained weights alone.
Limitations the authors surface themselves: the predicted-to-target BEV area ratio has a median of 1.4 but a P90 of 9.92, meaning a tail of predictions is almost 10× too large. Requiring the top candidate to also achieve BEV IoU ≥ 0.25 drops Hit@1 by about 5 points, so target identification is more reliable than box extent.
•
If you’re building a geo-aware agent on top of urban point clouds, this is evidence that an interactive BEV plus landmark lookup plus zoom tool interface is worth prototyping before investing in a heavy proposal-generation and ranking stack. The order-of-magnitude latency gap is especially relevant if you’d otherwise score thousands of candidates per query.
•
The GRPO recipe here uses a dense distance reward instead of a success threshold. If you’re training agents on tasks where early policies get zero overlap or zero exact-match reward, consider testing a similar smooth shaping term rather than relying on a sparse success bonus.
•
Treat the zero-shot STPLS3D-Refer and landmark-free numbers as suggestive, not conclusive. The authors generated those expressions with a VLM using OSM cues, and the landmark-free diagnostic covers only 30 manually verified queries. Worth replicating on an independently constructed benchmark before claiming the policy generalizes beyond landmark-anchored queries.
•
Code and checkpoints are released at GitHub under a research-only license that explicitly prohibits unconsented surveillance.
Urbex assumes you can build and calibrate a BEV plus view-to-world mapping for the scene and have landmark data available; a miscalibration can project a plausible-looking 2D box to the wrong world location. The reported ~10× inference speedup excludes offline preparation (BEV generation, calibration, landmark registration, and an optional ~2.3-hour oblique-view cache across the 12 CityRefer scenes); on-demand oblique rendering raises per-query latency to about 4.4 seconds. The five-run Hit@1 interval is based on decoding variation of one trained checkpoint, not independent training seeds, so it doesn’t bound training variability. Finally, failure analysis shows the agent still misidentifies targets when ten or more same-class distractors sit within 50 meters, so dense repeated-object scenes remain an open weakness.