Hunyuan3D-Buffalo 1.0 trains one model for 3D understanding, text-to-3D, instruction-guided editing, and part extraction by pairing a 3D vision-language model with a diffusion 3D generator, showing that scaling text-to-3D data alone (up to 50M pairs) unlocks editing capability without adding editing data.
Suppose you’re shipping a tool where an artist types “replace the head with a chicken head” on a 3D asset, and today you glue together three separate systems: one that parses the instruction, one that generates from scratch, and a third Nano3D-style pipeline that tries to edit in place. Each has its own failure mode, and unedited regions drift between hops. The dominant baseline for 3D editing is training-free latent surgery on a frozen generator like TRELLIS, which works but is unstable across objects and edit types. This paper argues you should train one model on all three tasks jointly, and shows they reinforce each other rather than compete.
The architecture has two collaborating modules. Hunyuan3D-VLM is a language model that reads colored point clouds through two channels, one for geometry (coordinates and normals), one for appearance (RGB), compressed via a Q-Former into 512 tokens and fused with text tokens. It also gets 133 new vocabulary tokens so it can emit quantized 3D bounding boxes as text, unifying captioning, grounding, and edit-instruction rewriting as sequence prediction. The generator is a Diffusion Transformer (DiT) initialized from Hunyuan3D-2.1, conditioned on VLM hidden states through a small MLP connector. For editing and part generation, the source object’s latent representation is concatenated with the noisy latent so the denoiser can see the original geometry directly. Training moves through four stages: VLM alignment, text-to-3D pretraining on 50M pairs, an “omni” stage mixing all tasks 1:1, then task-specific continued pretraining. All generative training uses Flow matching.
The editing-data problem was the real blocker, and their Nano3D-v2 pipeline is the answer:
def make_edit_pair(source_mesh, instruction):
views = render_8_canonical(source_mesh)
anchor = vlm_pick_best_view(views, instruction)
edited_2d = qwen_image_edit(anchor, instruction)
mask_2d = pixel_diff(anchor, edited_2d)
box_3d = learned_localizer(mask_2d, source_voxels) # AR transformer
edited_vox = trellis_flowedit(source_voxels, edited_2d, box_3d)
edited_vox = voxel_merge(edited_vox, source_voxels, box_3d) # freeze outside
mesh = lattice_refine(edited_vox); tex = natex_inpaint(mesh, box_3d)
return vlm_filter_and_relabel(source_mesh, (mesh, tex))
The prevailing assumption in 3D editing is that you need dedicated editing supervision to teach a model to edit. This paper shows the opposite. Editing capability emerges from scaling text-to-3D generation, because editing is bottlenecked by how well the generator can synthesize the requested target concept in the first place. The load-bearing evidence is a controlled experiment where adding 1,000 chicken samples to text-to-3D training, with zero new editing examples, is what finally lets the model perform “replace head with chicken head.”
•
The chicken-head experiment is the mechanism proof: text-to-3D data alone flipped a failed edit into a successful one, without touching the editing corpus. The paper reads this as license to scale the cheaper data source (generation pairs) instead of the expensive one (editing triplets).
•
Text-to-3D data scaling is monotonic and unsaturated. Human preference for overall quality climbed from 8.4% → 28.6% → 57.5% as pretraining data went from 3M to 15M to 50M samples, consistent across every rater.
•
On Edit3D-Bench, the 3D-VLM-conditioned variant cut average Chamfer Distance from Omni123’s 0.0684 to 0.0091 (an 86.7% reduction) and lifted F1 from Steer3D’s 0.2729 to 0.6515 (2.39× improvement). Swapping the CLIP text encoder for their 3D-VLM improved F1 from 0.6336 to 0.6515, so richer 3D-aware conditioning helps editing precision.
•
On UniPart-Bench, Hunyuan3D-VLM leads all prior 3D language models on both part-level Q&A (85.47 SBERT) and object captioning (72.94 SBERT).
•
Text-to-3D human preference: 55–57% across text alignment, geometry, and overall, versus 17.5% for the strongest baseline Omni123 (random-choice baseline is 25%).
Reach for the design when you’re building a 3D asset tool that needs to accept free-form language instructions (“add glasses,” “remove the wings,” “segment the wheels”) on user-supplied meshes, and you’re currently either regenerating from scratch or chaining a training-free editor onto a frozen generator. The takeaway from their ablation is that if editing quality is your goal, spend your data budget on generation pairs first: they’re much cheaper to synthesize than editing triplets, and editing rides on top.
The project page is linked but the paper doesn’t specify a code or weights release, so treat this as a design blueprint rather than a drop-in dependency. The Nano3D-v2 pipeline recipe (anchor-view selection, learned 3D box prediction, voxel merge to freeze unedited regions, LATTICE refinement, NaTex texture inpainting, VLM filtering) is described in enough detail to reproduce if you have the underlying components.
If editing is your product, invest in the generator, not the editor. The cheapest path to instruction-following 3D edits is a stronger text-to-3D backbone plus a mechanism that freezes the unedited region. Dedicated editing data helps at the margin, but the base model’s ability to imagine the target is what actually gates the result.
•
The chicken-head result is one qualitative example. “Generation improves editing” is compelling as a direction but the paper doesn’t quantify how much text-to-3D scaling substitutes for editing data across edit types.
•
Texture editing is out of scope. The current model edits geometry; texture editing lacks training data and remains an open problem the authors explicitly flag.
•
The Nano3D-v2 pipeline preserves geometry outside a predicted 3D box, but the authors note that regions inside the box that shouldn’t have changed can still drift, and this noise propagates into the trained model. The reported editing scores are bounded by that data-construction ceiling.