GaussianDream
A research project that teaches robots to manipulate objects by building 3D models of their environment using a technique called Gaussian Splatting. Code and paper are coming soon.
GaussianDream is a research project from a collaboration between Tuojing Intelligence and several universities including the Chinese Academy of Sciences, Tsinghua University, Beihang University, Carnegie Mellon University, and the University of Hong Kong. The project proposes a method for teaching robots how to manipulate objects by building a three-dimensional model of the world around them.
The core idea involves a technique called 3D Gaussian Splatting, which represents a scene as a collection of small, blurry blobs positioned in 3D space rather than as solid surfaces or polygons. This kind of representation can be generated quickly from visual input. GaussianDream uses this as the basis for a world model, meaning the system builds an internal picture of its environment that it can use to plan robot actions, not just perceive the scene.
The README is mostly a placeholder. The full paper, code, and model weights are listed as coming soon, so the repository does not yet contain runnable code. The acknowledgements mention that the policy backbone builds on prior work called OpenPI and pi-zero, and that the benchmarks used for evaluation are LIBERO and RoboCasa, which are simulation environments for testing robot learning.
If you want to follow this project, the README suggests starring the repository and checking back for the paper and code release. Contact information for the lead author is included for questions.
Where it fits
- Follow the research and wait for the code release to experiment with robot manipulation using 3D Gaussian Splatting world models.
- Use the LIBERO and RoboCasa simulation benchmarks referenced here to test your own robot learning policies.
- Build on the OpenPI and pi-zero policy backbones that this project extends for robotic manipulation tasks.