Microsoft’s Trellis.2 is an advanced open-source AI model that transforms a single photo into a detailed 3D model with physically based rendering materials using a novel sparse voxel (“ovoxel”) approach and transformer-based generation. While requiring substantial hardware and a complex setup, it enables rapid creation of high-resolution, PBR-ready 3D assets ideal for research and prototyping, though further refinement is needed for production use.
Microsoft’s Trellis.2 is an open-source AI model that converts a single photo into a full 3D model with physically based rendering (PBR) materials, including base color, roughness, metallic, and opacity layers. Developed by researchers from Microsoft, Singhua, USC, and Microsoft AI, this 4 billion parameter model can generate assets at resolutions ranging from 512³ to 1536³. Unlike traditional 3D generators that use smooth fields to define surfaces, Trellis.2 employs a novel “ovoxel” approach, storing only the cells of a 3D grid that the object touches, allowing it to handle open surfaces, non-manifold geometry, and internal structures more effectively.
The model compresses these large sparse grids using a sparse variational autoencoder (VAE), reducing the data to a manageable number of latent tokens that a standard diffusion transformer can process. This approach enables Trellis.2 to generate detailed 3D models from images efficiently. Performance-wise, the model runs faster on high-end GPUs like the H100, with generation times of a few seconds to about a minute depending on resolution. On a rented mid-range GPU (NVIDIA L40S), generation took longer—ranging from about one to two and a half minutes—but still produced impressive results with sharp surface details and consistent unseen backside geometry.
Running Trellis.2 requires significant hardware resources, ideally a GPU with at least 24 GB of VRAM, and a Linux environment. The setup process is complex, involving multiple dependencies, gated model weights from Hugging Face, and careful version management of libraries like PyTorch and OpenCV. The entire pipeline includes loading the environment map, running the model on the input image, rendering previews, and exporting the final 3D asset in GLB format. Exported meshes are initially very dense (up to 16 million faces) but are decimated to about 1 million faces with 4K textures for practical use.
While the output quality is impressive for a research release, the generated models resemble scan-style meshes rather than artist-optimized topology, meaning further retopology is likely needed for animation or production use. The project is currently intended for academic and research purposes, with no official releases or production-ready support yet. However, the relatively affordable cloud GPU rental cost (around $19/hour) makes it accessible for developers needing quick, rough 3D assets for games or prototypes, with each model costing roughly 7 to 10 cents to generate and export after the initial setup.
In summary, Trellis.2 represents a significant advancement in single-image 3D reconstruction, offering detailed, PBR-ready models from just one photo using innovative sparse voxel representation and transformer-based generation. Although the setup and hardware requirements are non-trivial, the results are promising for rapid asset creation in research and development contexts. Users interested in experimenting with Trellis.2 should be prepared for a technical installation process and consider it primarily a research tool until more polished releases become available.