What is the task?
Given a natural-language instruction, synthesize a complete indoor scene by orchestrating a heterogeneous set of tools for asset retrieval, layout generation, geometry processing, and rendering.
language→tool calls→3D scene
ECCV 2026
SceneOrchestra:
Efficient Agentic 3D Scene Synthesis via Full Tool-Call Trajectory Generation
Generating complete tool-call trajectories for controllable, realistic, and efficient 3D scene synthesis.
1University of Maryland, College Park
NATURAL-LANGUAGE BRIEF
“Design me a
living room with
a sofa, table, and lamp.”
living room with
a sofa, table, and lamp.”
LEARNED EXECUTION PLAN
retrieve_asset()layout_room()place_lighting()render_scene()
instruction → executable trajectory
01 · THE TASK
Turn natural language into a usable 3D world.
3D scene synthesis turns natural-language descriptions into environments for virtual reality, embodied AI, and robotic simulation.
Why does it matter?
High-quality, controllable scenes make virtual experiences more accessible and provide diverse, repeatable environments for training and evaluating embodied agents and robots.
VREmbodied AIRobotics
LIMITATIONMonolithic methods struggle to balance realism, controllability, and physical reliability;
Monolithic methods struggle to balance realism, controllability, and physical reliability;
agentic methods remain inefficient.
NON-AGENTIC
Single-paradigm trade-offs
Rule-based methods provide physical validity, data-driven methods capture realistic scene distributions, and LLM/VLM-powered methods offer flexible control. However, each follows a single paradigm and therefore struggles to achieve realism, fine-grained controllability, and physical reliability simultaneously.
AGENTIC
Two sources of inefficiency
- Next-step tool selection and parameter configuration are driven by heuristic rules, which can lead to suboptimal execution flows, unnecessary tool invocations, degraded output quality, and increased runtime.
- Rendering and reviewing intermediate results after every step introduces additional latency.
02 · THE METHOD
Plan the entire execution flow before running any tools.
SceneOrchestra maps each instruction directly to a complete tool-call trajectory. Inference therefore requires a single planning step followed by end-to-end execution.
Independent training
Stepwise SFT and DPO teach context-aware tool selection. Trajectory-level SFT and DPO teach the orchestrator to generate complete trajectories. In parallel, the discriminator learns to rank candidate trajectories.
Stepwise SFTTrajectory-level SFTStepwise DPOTrajectory-level DPO
Interleaved distillation
We first adapt the discriminator to the orchestrator’s evolving trajectory distribution, then distill its discrimination capability back into the orchestrator.
discriminator⇢orchestrator
One planned rollout
At test time, only the orchestrator is required. It generates a complete trajectory once and executes it end to end, without intermediate render–review loops.
instruction→full trajectory→scene
↗
Core idea: replace reactive, step-by-step orchestration with learned global planning while retaining the flexibility of a heterogeneous tool set.
03 · RESULTS
Better scenes with more efficient tool orchestration.
We evaluate SceneOrchestra through template instructions, complex instructions, and human studies, examining scene quality, instruction following, diversity, and generation efficiency.
QUALITATIVE COMPARISON
Across seen and unseen rooms
We evaluate ten room types—five seen and five unseen—using the same tool implementations as SceneWeaver. Our method produces more coherent layouts, richer details, and fewer subtle inconsistencies than prior open-vocabulary systems.
EXTRA RESULTS
Additional comparisons and diverse generations
For each room type, “Ours 1” and “Ours 2” show two different scenes generated by SceneOrchestra from the same template instruction.
QUANTITATIVE COMPARISON
Average over 10 room types · template prompt “Design me a <room_type>”
| Method | #Obj ↑ | #OB ↓ | #CN ↓ | Real. ↑ | Func. ↑ | Lay. ↑ | Comp. ↑ | Time (min) ↓ |
|---|---|---|---|---|---|---|---|---|
| LayoutGPT | 7.8 | 1.1 | 1.2 | 6.0 | 6.2 | 5.5 | 3.6 | 2.4 |
| Holodeck | 26.2 | 0.0 | 8.6 | 7.8 | 7.8 | 6.4 | 5.8 | 12.4 |
| I-Design | 12.3 | 0.3 | 0.1 | 6.9 | 7.6 | 6.3 | 4.5 | 6.1 |
| SceneWeaver | 27.0 | 0.0 | 0.2 | 8.8 | 8.7 | 7.2 | 7.5 | 91.0 |
| SceneOrchestra (ours) | 31.1 | 0.0 | 0.0 | 9.1 | 9.1 | 7.9 | 8.1 | 39.7 |
COMPLEX INSTRUCTIONS
Fine-grained control over objects and spatial relationships
We evaluate 70 instructions for unseen room types, each specifying object counts, positions, spatial relationships, and layout constraints. Each row pairs the complete instruction with outputs from SceneWeaver and Ours.
MEETING ROOM
“Generate a meeting room with a rectangular conference table at the center, surrounded by multiple chairs. Place a tall cabinet, a small side table, and a potted plant along the back wall.”
GARAGE
“Design a garage with a car parked at the center, two bicycles leaning against the back wall, and a workbench along the side wall.”
LAUNDRY ROOM
“Create a laundry room with two front-loading washing machines side by side against the back wall, a countertop with a folding area directly opposite, and a drying rack with hanging garments in the corner.”
BATHROOM
“Create a compact bathroom with a shower stall in the corner. Add a toilet, a sink with a small vanity and towel rack, and a bath mat on the floor.”
HOME OFFICE
“Design a home office with a desk and a computer monitor as the central workspace, paired with a chair. Add shelving units with books and small decorative items for storage and decoration.”
NURSERY
“Generate a nursery room with a baby bed and a small changing table for infant care. Add a rocking chair and a storage cabinet stocked with toys and baby supplies.”
OPEN-PLAN LIVING ROOM
“Generate an open-plan living room with a dining table surrounded by chairs on one side and a sofa with decorative pillows on the other. Include additional storage furniture such as a sideboard and a storage cabinet to complete the space.”
GAME ROOM
“Create a game room with a foosball table at the center, two bean bag chairs for casual seating, and an arcade machine against the wall.”
MEDICAL EXAMINATION ROOM
“Design a medical examination room with an exam table at the center and a rolling stool beside it. Include a sink and a glass-door medical cabinet.”
STUDIO APARTMENT
“Generate a studio apartment that combines a sleeping area and a small living/dining area in one open space. Include a bed, a sofa, a small dining table with chairs, and add a storage cabinet for everyday use.”
COMPLEX-SET AVERAGE
SceneOrchestra consistently outperforms SceneWeaver under detailed constraints.
| Method | #Obj ↑ | #OB ↓ | #CN ↓ | Real. ↑ | Func. ↑ | Lay. ↑ | Comp. ↑ | Time ↓ |
|---|---|---|---|---|---|---|---|---|
| SceneWeaver | 24.3 | 0.2 | 0.4 | 8.2 | 7.6 | 6.7 | 7.3 | 82.6 |
| SceneOrchestra (ours) | 27.6 | 0.1 | 0.2 | 8.6 | 8.3 | 7.2 | 7.7 | 46.9 |
HUMAN EVALUATION
Human judgments support the automatic evaluation
Sixteen participants rated 30 scenes on a 1–10 scale. In a separate pairwise study, 18 participants compared SceneOrchestra with each baseline in terms of overall preference and scene diversity.
Human rating · 1–10 scale
| Method | Real. | Func. | Lay. | Comp. |
|---|---|---|---|---|
| LayoutGPT | 3.01 | 2.97 | 2.84 | 3.25 |
| Holodeck | 5.57 | 5.78 | 5.29 | 5.88 |
| I-Design | 4.73 | 4.98 | 4.33 | 4.64 |
| SceneWeaver | 6.72 | 6.83 | 6.41 | 6.57 |
| SceneOrchestra | 7.60 | 7.99 | 7.46 | 7.76 |
Pairwise study · preference / diversity
| Compared with | Preference | Diversity |
|---|---|---|
| LayoutGPT | 100% | 100% |
| Holodeck | 84.4% | 88.9% |
| I-Design | 95.6% | 96.6% |
| SceneWeaver | 76.7% | 80.8% |
✓
Takeaway: people preferred SceneOrchestra’s scenes and found them more diverse in the vast majority of pairwise comparisons.
04 · CITATION
Build on SceneOrchestra.
If this work helps your research, please cite:
@article{he2026sceneorchestra,
title={SceneOrchestra: Efficient Agentic 3D Scene Synthesis via Full Tool-Call Trajectory Generation},
author={He, Yun and Yu, Kelin and Zwicker, Matthias},
journal={arXiv preprint arXiv:2604.19907},
year={2026}
}