Get Started
Home
Topics
Search
Library
6 min read · Agents · Code Generation · Added Oct 17 · Paper published Oct 15, 2026

3DCodeVerse: Exploring 3D Worlds as Code

Source: research paper via Hugging Face Daily Papers
3D asset generation forces a choice between uneditable mesh blobs and brittle one-shot code that crashes or clips. 3DCodeVerse closes the loop: a planner-DAG-execute-judge harness with deterministic geometric vetoes over VLM critique wins 90% of human comparisons against a same-backbone commercial baseline.
TL;DR
3DCodeVerse generates 3D content as executable programs and refines them with a multi-agent loop that runs the code, measures the geometry, and judges renders, producing fixes without human review. Human raters prefer its outputs over Claude Design in 90% of head-to-head object comparisons when both use the same backbone.
Why It Matters
If you want an AI to make a 3D asset (a chair, a hinged laptop, a scene for robot training), you have two bad options today. Mesh-generation models give you an opaque blob you cannot edit cleanly. Letting an LLM write 3D code (a Blender Python (bpy) script, a CadQuery program, a Three.js scene, a GLSL shader) gives you something editable and inspectable, but one-shot code generation fails often: the script crashes on an API that changed, legs float above the seat, hinges clip through the body, or the geometry just looks wrong.
Prior work has pieces of a solution. SceneCraft and VIGA use rendered images as feedback. Articraft adds a validator for articulated assets. 3DCodeBench benchmarks text-to-Blender tasks. But each targets one slice (one language, one asset type), and no shared corpus exists for training open models to write executable 3D code across environments. The paper’s premise is that the field needs a joint dataset plus a reusable verification harness so the generation loop can close on its own.
How It Works
There are two artifacts. The first is 3DCodeVerse-2M, a corpus of roughly 270K unique executable 3D programs (expanded via caption variants and per-pass shader rows to ~2M training rows) covering CAD, procedural Blender, OpenSCAD, URDF articulated mechanisms, Three.js scenes, and GLSL shaders. The authors build it by curating existing sources (translating DeepCAD command sequences into CadQuery, inlining Articraft geometry helpers), harvesting Shadertoy shaders, and generating new programs with their own harness. Every program is bundled with its assets so it actually runs in a pinned runtime.
The second is the 3DCodeVerse Harness, which treats 3D code generation as a verify-and-repair loop. A planner LLM reads the user prompt and produces a structural plan: named parts with bounding boxes, materials, attachment relations, and an acceptance checklist. This plan becomes a Directed Acyclic Graph (DAG) of coding and assembly jobs that sub-agents can work on in parallel because each job owns disjoint files. The resulting program is executed, meshes are measured for contacts, collisions, dimensions, and joint sweeps, and a VLM judge inspects multi-view renders against the checklist. All three feedback types (execution errors, geometric measurements, visual critique) are compiled into prioritized revision tasks that a fix agent applies.
Two design details matter. First, deterministic geometric checks veto contradictory visual complaints: if the connectivity gate confirms parts are touching, the judge cannot claim they are floating. Second, during development the authors alternate between tuning the judge (holding generation fixed, scoring a bank of stored artifacts for consistency) and tuning the generation workflow (holding the judge fixed). This separates “the judge got noisier” from “the generator got worse.”
plan = planner(prompt, env) # parts, bboxes, checklist dag = build_file_scoped_dag(plan) program = run_subagents(dag) # concurrent file-disjoint coding for t in range(T): # T=4 by default artifact, log = execute(program, env) measurements = geometric_gates(artifact) # contacts, collisions, dims renders = multiview_render(artifact) verdict = vlm_judge(prompt, plan, measurements, renders) tasks = prioritize(gate_errors=measurements, checklist=verdict, visual=verdict) if not tasks: break program = fix_agent(program, tasks) return best_round_by_score()
What They Found
Fine-tuning on 3DCodeVerse-2M dramatically improves whether open models produce code that runs at all. Zero-shot Qwen3.5-9B executes 0 of 212 3DCodeBench tasks because it calls Blender scene-clearing operators that no longer exist in Blender 5. After LoRA Supervised Fine-Tuning, the same model executes 84.9%. The 27B variant with SFT plus Direct Preference Optimization reaches 92.9% execution on 3DCodeBench and 98.5% on CadQuery. Geometric fidelity rises too: F@0.05 (a mesh-overlap score) on 3DCodeBench goes from 0.000 to 0.317 for the 9B model.
For the harness, the headline comparison swaps only the surrounding workflow while fixing the backbone LLM. On 100 text-to-3D-object prompts, outputs from each model’s native coding harness are preferred in only 11\u201331% of human comparisons versus the same model inside 3DCodeVerse Harness. Against Claude Design, a product that also uses Claude Opus 5.5, 3DCodeVerse Harness wins 90% of pairwise human comparisons when both run on Opus 5.5. The pattern extends to articulated objects, scenes, and graphics, with human preference between 78\u201388%. Across backbones on hard prompts, the harness lifts judged quality by 15.5 to 29.1 points over one-shot generation.
Two caveats on the evidence. The judge is itself a VLM (Gemini 3.1 Pro), but human preferences track the judge’s scores, and the authors measure judge noise (standard deviation of 0.030 on a 0\u20131 scale) as much smaller than the system-level gains. The harness is not free: a typical object run costs about $3.31 per prompt, driven by multi-round refinement and tool calls, though prompt caching recovers ~94% of input tokens.
What’s Useful
If you need editable 3D assets with explicit geometry (robotics simulation targets, controllable video-generation references, parameterized CAD parts), code-as-asset is a reasonable path and this paper gives you a working recipe. The articulated-object demo passes a door-and-handle geometry check in 12 of 12 rollouts, versus 7 of 36 for a video world-action model baseline, suggesting that programmatic assets preserve rigid structure better than latent rollouts for simulator-ready content.
If you fine-tune open models for domain-specific code generation, the dataset-construction pattern is worth copying even outside 3D. Running a verification harness as a data engine (keeping successful programs, failed attempts, and targeted repairs as supervision) gave them execution-verified bootstrap programs that measurably improved target-dialect executability in mixtures.
If you build agent systems with iterative refinement, two harness-level findings are worth testing in your own loop. Returning a semantic “fail” verdict as a valid tool observation (rather than as a protocol-level error) cut redundant retry loops by 20x in their telemetry, because commercial agents stop treating negative evidence as a transport failure. Separating judge tuning from generator tuning on a fixed bank of artifacts also isolates whether a regression came from the critic or the actor.
The authors say they will release datasets, harness, models, and training recipes. The paper does not provide a repository URL in the supplied text.
Caveats
The win-rate numbers compare against each model’s default coding harness (Codex CLI, Claude Code, Gemini CLI, Claude Design), which are general-purpose and not tuned for 3D. The gain is “specialized 3D harness beats generic coding agent on 3D tasks,” not “this method beats a competing 3D-specialized system.” The judge is a VLM, and even with deterministic geometric caps and human-preference corroboration, both evaluator and in-loop judge are from the same model family (Gemini), so some shared bias is possible. Execution rates are measured in pinned runtimes; drift in Blender or library versions will shift the numbers. Fine-tuning results are on Qwen models with LoRA; the paper does not test whether the data mixture transfers to other base-model families.
Topics
Agents
Code Generation
Multimodal
Computer Vision
Agents
Code Generation
Multimodal
Computer Vision
Up next in Agents
DeltaStreamer: Towards Human-Like Agentic Streaming Video Understanding
MAI-Gym: An Open All-in-One Platform for GUI Agent Evaluation and RL Training across Mobile and Computer Environments
Don't miss new content
Log in to follow topics and personalize your feed.
Related topics you might like
Agents288 episodes
Multimodal153 episodes
Code Generation67 episodes