AI agents build 3D scenes from photos but have no idea if they got it right
decoder ·
Researchers from the University of Maryland and AWS have introduced LEGO-Anything, a system that uses coding agents to generate editable 3D Blender scenes from single photos through an iterative 'Image-to-Code' process. To evaluate these agents, the team created LEGO-Bench, a benchmark utilizing simulator-rendered images that provide hidden ground truth for precise scoring of validity, geometry, and appearance. While all tested GPT configurations produced usable scene artifacts, geometric accuracy varied significantly, with the best model achieving only 53.4% on indoor scenes. A critical finding was that the agents lack reliable self-assessment capabilities, often failing to recognize when their own revisions degraded the scene's quality. In response, the researchers developed LEGO-Plugin, which replaces subjective judgment with concrete measurements and successfully improved performance across all models. Despite these advances, the reconstructed scenes currently offer only modest results for downstream vision tasks like object detection and depth estimation compared to specialized AI models. The study highlights that while executable 3D programs are promising, significant gaps remain between functional outputs and faithful geometric reconstructions.
decoder