shap-e
Generate 3D objects conditioned on text or images
Shap-E is an OpenAI research model that generates 3D objects from text prompts or images, outputting shapes as implicit functions that can be previewed as animated 3D models.
Shap-E is a research project from OpenAI that generates three-dimensional objects from text descriptions or images. You can type a prompt like "a chair that looks like an avocado" or "a birthday cupcake" and the model produces a 3D shape. You can also provide a photograph or rendered image and have the model generate a 3D version of the object shown.
The approach is based on a research paper and works by learning a compact mathematical representation of 3D shapes, called implicit functions, which can be decoded into viewable 3D models. This is different from producing a mesh or a point cloud directly. The outputs can be displayed as rotating animated previews.
Installing the library requires Python and pip. The repository includes three Jupyter notebooks to help people get started: one for text-to-3D generation, one for image-to-3D generation, and one that demonstrates encoding an existing 3D model back into the learned representation. The image-to-3D path works best when the background is removed from the input image first. The encoding notebook additionally requires Blender, a free open source 3D software application, to generate the image renders it needs as input.
This is a research release, meaning it is intended to share the methods and models described in the accompanying paper rather than to serve as a finished product for end users.
Where it fits
- Generate a 3D model of any object from a text description like 'a chair shaped like an avocado'.
- Convert a photograph into a 3D shape for use in design or game prototyping.
- Experiment with encoding existing 3D models back into the Shap-E learned representation.