ToolArtist post-trains a Unified Multimodal Model so that reasoning, web/image search, and native image generation are all actions of a single policy, rather than a fixed pipeline or a search agent that hands its prompt to a separate generator downstream.
You’re building a product that generates images from user prompts like “a saltpeter miner at the port of Antofagasta in February 1891.” Today’s stack is usually: an LLM agent does web search, writes a fat descriptive prompt, then calls Flux or Nano Banana. The agent stops the moment it hands off. It never sees the image it caused, so it can’t notice the clothing is wrong and search again.
The dominant alternative in the literature, Unify-Agent, uses a unified multimodal model but still runs a prescribed “understand, search, recaption, generate” pipeline. In both cases, when to draw is decided by the scaffold, not the model.
The base model is Emu3.5, a Unified Multimodal Model that emits text tokens and image tokens in the same autoregressive stream. ToolArtist treats “call TextSearch,” “call ImageSearch,” and “draw an image now” as three action types the model chooses between at every step, following the ReAct pattern. Because a drawn image lands back in the context as visual tokens, the model can inspect its own output, decide it got the Socotra coastline wrong, search again, and redraw.
Getting the model there takes two stages. For Supervised Fine-Tuning, a teacher agent (given real text search, image search, and an external generator, gemini-3-pro-image-preview) produces 7,132 multi-turn trajectories. A converter then rewrites each external-generator call into a native visual-caption span followed by image tokens, so during training the model learns “I generated this image myself” rather than “I called a tool that returned an image.” A loss mask supervises only tokens the policy would emit, never the retrieved evidence.
Stage two is Reason-Act-Draw GRPO, their variant of Group Relative Policy Optimization (GRPO) for these interleaved trajectories. Per prompt they sample B trajectories, score each with two complementary rewards, and apply the group-relative advantage to every policy token in the trajectory:
for prompt in batch:
trajs = [rollout(policy, prompt, env) for _ in range(B)]
for h in trajs:
r_intent = judge_caption(h.final_caption, prompt, h.evidence)
r_quality = world_knowledge_rm(h.final_image, prompt, h.final_caption)
r = 0.5*r_intent + 0.5*r_quality
r += format_ok + draw_signal - length_pen - no_draw_pen
A = groupwise_zscore([h.reward for h in trajs])
update(policy, trajs, A) # clipped ratio, KL to SFT ref
Intent reward asks “is the final caption a good generation prompt given what you searched?” Quality reward, from a world-knowledge reward model, scores the pixels on faithfulness, visual correctness, text accuracy, and aesthetics (weights 0.1/0.4/0.4/0.1). Auxiliary terms exist because without a no_draw_pen, the policy collapses into a search-only mode that never emits an image.
The prevailing assumption in agentic image generation is that a smart search-and-planning agent can front-load all reasoning into a great prompt, then hand that prompt to a specialist generator. This paper argues the opposite. When drawing is an action the same policy chose, the model can see its own image land in context, notice it’s wrong, and re-search or redraw. Delegating generation ends the feedback loop at the moment it becomes most useful. The evidence to watch is the head-to-head on WorldGenBench Humanities against Unify-Agent and GenSearcher, both of which use search agents but delegate the pixels.
The load-bearing result is the head-to-head against other agentic systems on WorldGenBench Humanities, which scores images against per-prompt checklists of concrete world-knowledge items (correct clothing, local livestock, regional architecture). ToolArtist averages 22.10 KCS versus 15.58 for Unify-Agent and 13.66 for GenSearcher-Qwen-Image. On the Socotra dragon’s-blood case, ToolArtist hits 0.625 where Unify-Agent scores 0.000 and GenSearcher 0.250, because it correctly places grazing goats, traditional tools, storage vessels, and a coastline in one scene.
•
On WISE benchmark, overall score 0.79, ahead of the other agentic image-generation systems in their table. Strongest proprietary models still lead on Time and Space categories.
•
Ablating source-aware summaries on the image-search tool drops WISE overall from 0.79 to 0.61, with Biology falling by 0.50. Grounding a reference image to its source page matters most when identity, not appearance, is what has to be right.
•
The Reason-Act-Draw GRPO curve rises from an initial dip to ~0.41 by end of training with no entropy collapse. The paper does not report an SFT-only vs SFT+RL headline number in the excerpt.
On a per-continent split of WorldGenBench, ToolArtist wins Africa, Antarctica, and Asia; Qwen-Image wins Europe, North America, and Oceania; Unify-Agent wins South America. It is not a clean sweep.
Reach for this design when you’re shipping a system where the generator needs facts it doesn’t know, and where a checklist-style verifier exists or can be built (product spec compliance, brand-guideline adherence, historically or culturally grounded scenes). The move is: don’t hand a fat prompt to Flux and walk away. Keep the generator inside the same policy that did the search, so a failed draft can trigger another search round instead of a full user-visible retry.
The authors say they release the 7,132-trajectory SFT dataset and the full SFT + RL infrastructure for Unified Multimodal Model agents. The excerpt doesn’t include a repo URL or license, and doesn’t state whether ToolArtist weights themselves are released. Directly reusable pieces: the trajectory-conversion trick (rewrite external-tool image calls into native caption-plus-visual-token spans) is applicable to any Unified Multimodal Model you want to teach agentic behavior without retraining a generator, and the dual intent+quality reward is a reasonable template for RL on any generate-and-verify task.
If your agent can see its own output, don’t hand generation off to a tool it can’t see through. Delegating the last step means the model never learns that the picture was wrong, only that its prompt sounded right.
•
The headline win is on knowledge-heavy, checklist-scored benchmarks. On visual quality alone, frontier proprietary generators still lead several WISE categories, and Qwen-Image, without any agent loop, matches or beats ToolArtist on three of seven WorldGenBench continents.
•
The whole approach presupposes a Unified Multimodal Model strong enough to natively generate competitive images. On top of a weaker generator, keeping generation in-policy would just mean worse pixels with extra steps.
•
The quality reward is a learned world-knowledge reward model whose training and calibration aren’t detailed in the excerpt. RL against a reward model can chase whatever that model rewards, which may not fully generalize beyond WISE/WorldGenBench-style checklists.