Benchmarking Spatial Reasoning of LLMs in Text-to-3D Modeling

Abstract: Large Language Models (LLMs) have achieved significant success in software engineering. However, their capacity for 3D spatial reasoning and procedural modeling remains underexplored. While current text-to-3D methods are effective at generating visual representations such as meshes or point clouds, these outputs usually lack parametric editability and underlying construction logic. As a result, generating procedural Computer-Aided Design (CAD) models directly from text remains a complex and largely overlooked problem. This thesis evaluates the 3D spatial reasoning capabilities of open-weight LLMs in order to address this challenge.

A fully automated benchmarking framework was created to measure the generation of procedural OpenSCAD geometry based only on natural language descriptions. The evaluation uses a balanced dataset of 900 3D designs with reference source code and structural metadata for different complexity levels. Before assessing geometric performance, a two-stage spatial alignment process is applied. This process uses Principal Component Analysis (PCA) and Iterative Closest Point (ICP) registration to evaluate geometries on a coordinate-independent basis.

Furthermore, a new Composite Success Score ($S_c$) is introduced. It combines volumetric overlap, surface similarity, and syntactical compilation validity into one metric. By comparing state-of-the-art models, including Qwen2.5-Coder:32B, Codestral:22B, and DeepSeek-Coder-V2-Lite, using Zero-Shot and Chain-of-Thought prompting strategies, this research proposes a robust and scalable methodology for evaluating true spatial reasoning in generative AI.

COMPLETED