ReDesign turns a raster design into an editable layer tree by having a VLM agent expand one node at a time, with a verifier that accepts, prunes, or retries each split locally, cutting error cascades enough to run ~7.1× faster than serial tool chains on the same tools.
You’ve shipped a product that lets marketing teams tweak a poster: change the headline, swap the color theme, nudge the logo. The source Figma file is lost. All you have is a PNG. Today someone rebuilds the layers by hand in Illustrator before any edit is possible. Generative image editors like Nano Banana 2 can retouch the PNG directly, but the paper shows they drift on spatial edits (they resize the canvas, they change unrelated regions). The dominant research alternative is layered decomposition, e.g. LayerD or Qwen-Image-Layered, which splits a raster into RGBA layers but doesn’t recover text as text, shapes as vectors, or a correct z-order. ReDesign targets the gap between “pixels that look right” and “a file whose knobs behave when you turn them.”
The core reframing: don’t run tools in a fixed sequence, grow a tree that mirrors how a designer actually structures a file. The root node is the whole image. At each step, a controller looks at one node (a cropped region plus its history) and picks one action from a fixed toolbox: OCR-then-inpaint for text, layered decomposition, connected-component splitting, detect-plus-segment-plus-inpaint for entangled objects, or vectorization at the leaves. The proposed children go through graceful verification before they’re committed. The verifier asks two concrete questions: do the children together cover the parent region, and does any child hallucinate content beyond the parent? Based on that, it emits accept, prune the bad children, or retry the same parent with a different tool or setting. Because each expansion only depends on its parent’s lineage, sibling nodes expand in parallel. Rough control flow:
frontier = [root(image)]
while frontier:
node = frontier.pop() # runs in parallel across leaves
action = controller.choose(node.crop, node.history)
children = run_tool(action, node)
verdict = verifier(node, children) # accept | prune | retry
if verdict == "retry":
node.history.append(failure); frontier.append(node)
else:
keep = verdict.accepted_children
frontier.extend(c for c in keep if not is_leaf(c))
return export_json(root)
The controller and verifier are both a VLM (Gemini-3-Flash by default, with GPT-5 mini as a tested swap); no training is done, tools are used off the shelf.
The prevailing pattern for tool-using agents is a serial trajectory with an end-of-run check, in the spirit of ReAct or Reflexion: run the chain, evaluate the final output, retry on failure. This paper argues the opposite for structural reconstruction. When each tool call permanently changes the state the next tool sees, correctness has to be enforced at every expansion, not at the end. Many small local checks are cheaper than a few big terminal ones because they kill error cascades before deeper branches are built on top of them. The load-bearing evidence isn’t the headline benchmark, it’s the cost-accuracy plot where step-level verification is simultaneously faster, more accurate, and lower-variance than terminal verification on the same pipeline.
•
Step-level verification dominates terminal verification on the same tools: higher PSNR, lower run-to-run variance in tool-call counts, and shorter wall-clock. This is the finding that makes the thesis load-bearing.
•
Tree-shaped expansion enables parallel node processing, yielding ~7.1× speedup over a serial ReAct-style agent with the same tools and memory.
•
On the new Figma Edit Replay Benchmark (909 files, 14,796 edits spanning layout, color, and text), ReDesign posts the highest edit-replay SSIM across all edit types and the highest text recall after text edits. The serial tool agent degrades sharply on text because early cascade errors corrupt glyphs before OCR ever runs.
•
On the Crello dataset dataset, layered baselines close the gap on global appearance metrics (closer training distribution), but ReDesign still leads on layout fidelity (Panoptic Quality (PQ) and detection F1) without training on Crello.
•
Backbone swap: replacing Gemini with GPT-5 mini shifts a coverage-vs-fidelity trade-off but stays state-of-the-art, so the structural workflow, not the specific VLM, is doing the work.
•
Behavior analysis: the controller learns a coarse-to-fine pattern on its own. Text extraction dominates at shallow depths, layered decomposition at depth 1, connected-component splitting at depth 2, detect-and-segment deeper down.
Reach for this when you’re building a “re-editable asset” feature: a user uploads a screenshot or a flattened export and expects to change copy, recolor a brand palette, or reposition a logo without regenerating pixels. The concrete swap vs. a serial agent is to organize the decomposition as a tree of regions and attach a lightweight verifier that answers only two questions per expansion (coverage and hallucination), so a bad OCR call or a bad segmentation gets retried on its own subtree instead of poisoning the rest of the reconstruction.
Artifacts: there’s a project page with interactive demos. The paper does not link a code repository, and the Figma Edit Replay Benchmark is described but release terms aren’t stated in the text. The system is training-free and orchestrated with LangGraph, relying on off-the-shelf tools (VTracer for vectorization, Segment Anything Model for segmentation, Grounding DINO for open-set detection, LaMa for inpainting, plus a commercial VLM API); the authors note the local tools fit under 8GB of VRAM with standard optimization.
When each tool call mutates state the next tool depends on, verify at every step, not at the end, and shape the agent’s memory as a tree so failure stays local and repair stays cheap.
•
The “right” granularity for an editable file is subjective. ReDesign doesn’t train to match Figma’s exact layer splits, so IoU-based matching in the edit-replay protocol can penalize reconstructions that render correctly but group elements differently than the ground truth.
•
Reliability rests on the verifier’s two checks (coverage and hallucination) being well-posed. For photographic regions or heavily entangled compositions where “does this cover the parent” is fuzzy, the local accept/prune/retry signal gets noisier and the cascade-prevention argument weakens.
•
Cost is not free: each expansion pays for a VLM controller call plus a VLM verifier call. The 7.1× speedup is against a serial agent using the same commercial VLM, not against a purpose-built end-to-end model, and the paper doesn’t report absolute dollar or token cost per reconstruction.