Physically Compatible 3D Object Modeling from a Single Image

Minghao Guo1, Bohan Wang1, Pingchuan Ma1, Tianyuan Zhang1, Crystal Elaine Owens1, Chuang Gan2,3, Joshua B. Tenenbaum1,4,5 Kaiming He1 Wojciech Matusik1
1MIT CSAIL, 2UMass Amherst, 3MIT-IBM Waston AI Lab, 4MIT BCS, 5Center for Brains, Minds and Machines
arXiv Code (Coming Soon)

PhysComp transforms single images into 3D physical objects that withstand real-world physical forces.

Abstract

We present a computational framework that transforms single images into 3D physical objects. The visual geometry of a physical object in an image is determined by three orthogonal attributes: mechanical properties, external forces, and rest-shape geometry. Existing single-view 3D reconstruction methods often overlook this underlying composition, presuming rigidity or neglecting external forces. Consequently, the reconstructed objects fail to withstand real-world physical forces, resulting in instability or undesirable deformation -- diverging from their intended designs as depicted in the image. Our optimization framework addresses this by embedding physical compatibility into the reconstruction process. We explicitly decompose the three physical attributes and link them through static equilibrium, which serves as a hard constraint, ensuring that the optimized physical shapes exhibit desired physical behaviors. Evaluations on a dataset collected from Objaverse demonstrate that our framework consistently enhances the physical realism of 3D models over existing methods. The utility of our framework extends to practical applications in dynamic simulations and 3D printing, where adherence to physical compatibility is paramount.



(Left) The visual geometry of a physical object in an image is determined by three orthogonal attributes: mechanical properties, external forces, and rest-shape geometry. (Right) Given predefined mechanical properties and external forces, our pipeline optimizes the rest-shape geometry to ensure that the shape, when in a state of static equilibrium, aligns with the target image and meets stability criteria.

Physical Compatibility Optimization

Soft Objects

Rest shapes optimized using our approach result in static shapes that closely match the input images when subjected to gravity. In contrast, shapes without the optimization fail to replicate the geometry in the input image.

Input Image
Wonder3D
MeshLRM
TripoSR
LGM
TetSphere
rest w/o optim
static w/o optim
rest w/ optim
static w/ optim

Stiff Objects

Our optimization process ensures that the optimized shapes are capable of supporting themselves, whereas the baseline methods fail to achieve this stability.

Input Image
Wonder3D
MeshLRM
TripoSR
LGM
TetSphere
w/o optim
w/ optim
w/o optim
w/ optim

Varying Materials

By changing the material properties, PhysComp can produce various rest-shape geometries, which all result in the same static shapes. Although these static shapes appear identical under static equilibrium, they exhibit different deformation when subjected to the same external forces, attributable to the differences in their material properties.

Applications

Dynamic Simulation

Results of PhysComp are simulation-ready and can be seamlessly integrated into dynamic simulation pipeline to produce complex dynamics and motions.



3D Printing

Real-world validation using 3D printing shows that shapes optimized using PhysComp closely replicate the input images.



BibTeX

@article{guo2024physcomp,
  author    = {Guo, Minghao and Wang, Bohan and Ma, Pingchuan and Zhang, Tianyuan and Owens, Crystal Elaine and Gan, Chuang and Tenenbaum, Joshua B. and He, Kaiming and Matusik, Wojciech},
  title     = {Physically Compatible 3D Object Modeling from a Single Image},
  journal   = {arXiv preprint arXiv:2405.20510},
  year      = {2024},
}