DroneCATS drops a swappable Multimodal Large Language Model straight into a drone’s control loop with a four-action interface declared only in the prompt, and finds that small open models fly into the 5 m success radius more often than frontier models but lose episodes by declaring arrival prematurely or never at all.
You’ve shipped an agent that calls tools in a loop: browse, click, submit, then a done tool that ends the task. Your logs show the agent often reaches the right page but either fires done on the wrong screen or keeps browsing forever. That’s the drone version of the same bug, and the paper isolates it.
Recent drone-plus-LLM systems like See, Point, Fly (SPF) and Fly0 have quietly narrowed what the model does, handing trajectory generation, verification, or termination-by-distance-threshold to non-learned modules. The paper argues this hides where models actually break, and rebuilds the setup so the model owns searching, deliberating, and declaring done.
DroneCATS-Agent gives the model four actions, described only in the system prompt, no fine-tuning, no function-calling schema:
•
go with a normalized image point and a monocular depth estimate. A rule-based controller back-projects the pixel through a pinhole camera model into a body-frame velocity setpoint.
•
rotate yaws in place so the model can search when the target is out of frame.
•
think hovers the drone and enables extended reasoning for the next step only. Every token costs latency in a closed loop, so deliberation is opt-in.
•
finished is the model’s own claim of arrival. A verifier checks it against ground truth afterwards. No external distance threshold ends the episode.
That last piece is the load-bearing design choice. Prior systems terminate when the drone crosses a distance the model never sees. Here, termination is a token the model must emit, and the episode keeps running if the first declaration is wrong, so later declarations still count.
while t < 300:
frame = drone.egocentric_rgb()
action = mllm(prompt, frame, last_5_actions)
if action.type == "go":
dx, dy, dz = backproject(action.point, action.depth_m)
drone.velocity_setpoint(dx, dy, dz, hold=2.5s)
elif action.type == "rotate": drone.yaw(action.angle_deg)
elif action.type == "think": drone.hover(); enable_reasoning_next_step()
elif action.type == "finished":
log_declaration(pose, target_visible, dist_to_target)
drone.hover() # episode keeps running
The DroneCATS benchmark benchmark then varies two binary axes: target moving or not, target visible in the first frame or not. One success rule covers all four cells: at least one declaration within 5 m while the target is visible. A commanding suite hands one model four drones and four views in a single context, with four look-alike candidates distinguishable only up close.
The prevailing pattern in Vision-Language-Action model drone stacks is to shrink the model’s role until it’s just a grounding module, and let external logic decide when the task is done. This paper shows the opposite. When you give the model the whole loop, including the action that ends it, the thing that separates a 2B open model from a frontier model is not perception or flying but the discipline to keep the declared action space intact across a long episode. The evidence is the gap between entering the 5 m radius and successfully declaring arrival, not the headline success rates.
On the approaching cell, Qwen3.5-9B enters the 5 m success radius in 90% of episodes, more often than any frontier model, and converts only 35% into successes. It declares arrival on average at 0.63 of the start distance, i.e. well before it has closed. Qwen3.5-2B goes the other way: it declares in 25% of episodes at 1.28 of the start distance (further out than it started), and never succeeds. Cosmos3-Edge-2B enters the radius in 25% of episodes and never declares once. This is the load-bearing finding: navigation holds, the terminating action does not.
Secondary results in support:
•
The best model, Gemini 3.7 Flash, solves only 13 of 20 approaching episodes. Withholding the target from the first frame drops the best score to 8 of 20, and no model exceeds that.
•
The Qwen3.5 family is monotone in scale across the four cells: 33.8, 21.3, 12.5, 0% for 27B, 9B, 4B, 2B.
•
The embodiment-specialised Gemini Robotics-ER 2 averages 47.5% across cells vs 57.5% for its generalist sibling. Specialisation buys nothing visible here.
•
In fleet commanding with four drones in one context, Qwen3.5-9B emits the identical pixel and depth for all four different views in 70% of its all-go steps (27B: 58%), so at most one of the four commands can be grounded. Frontier models sit at 0-1%.
•
One-drone rankings do not predict commanding: GPT-5 goes from 60% single-drone to 20% commanding, while Gemini 3.7 Flash goes 65% → 80%.
•
A three-flight replication of Gemini 3.7 Flash shows per-cell standard deviations of 9-13 points, matching the binomial expectation for 20 episodes. Tier structure survives; neighbour orderings inside a tier should not be over-read.
•
The think action action was invoked 1,797 times. On step-level matched comparisons it did not move outcomes: the three steps after a think gained only +0.03 m/step in closing rate over ordinary steps.
Reach for this framing when you’re evaluating an agent that owns its own termination, whether that’s a coding agent calling done, a browser agent submitting a form, or an embodied policy claiming task completion. The paper’s move is to stop treating “did it reach the goal region” as the metric and start scoring “did it correctly claim it reached the goal region”. If your logs show high task-radius entry and low success, you probably have the same declaration bug, and no amount of grounding improvement will fix it.
Code and benchmark are promised at GitHub. The benchmark ships 100 episodes (80 single-drone across two Unreal maps in AirSim, 20 commanding at N=4), a per-episode JSON spec, and Algorithm 1 as the sole scoring rule so radius δ can be changed post-hoc without re-flying. The prompt is given verbatim in the appendix, which matters because the entire action space lives there.
If your agent owns its own done, score whether it says done correctly, not whether it wandered into the right place. Termination-by-external-threshold is the reason prior drone benchmarks made backbones look interchangeable. Once the model has to emit the terminating token, small open models fall apart on protocol discipline long before they fall apart on perception or control.
•
Everything runs in AirSim with SimpleFlight, not on hardware. The paper flags that host-load-induced control-rate jitter already moves success rates, so real onboard compute at ~2 Hz is a genuine open question, not a formality.
•
The single-drone suite is 20 episodes per cell, so per-cell standard deviation is ~9-13 points. Adjacent models in the leaderboard sit inside this noise; only the tier gaps (frontier vs small open) and the reached-vs-declared decomposition are robust.
•
The 2B and 4B open models were served locally with schema-constrained decoding, while frontier models decoded freely. The authors show only 2 of many thousands of free-decode replies failed to parse, so they attribute the gap to protocol adherence rather than formatting, but the serving asymmetry is real and could interact with think in ways the step-level analysis cannot fully isolate.