3D Gaussian Splatting reconstructs a scene from a set of images or a video by representing it as a cloud of coloured Gaussians, each with a position, a covariance, a colour and an alpha, then optimising them through a differentiable rasteriser until the rendered views match the captured photographs.
3D Reconstruction with 3D Gaussian Splatting
In the rapidly evolving landscape of 3D modeling, innovations consistently push the envelope, enhancing our capabilities to create, visualize, and interact with digital objects.
One such groundbreaking technique that’s making waves is 3D Gaussian Splatting. Though rooted in decades-old principles, this method leverages modern computational power to transform how we handle and render 3D data from simple 2D images or videos.
In this Live Course Session, we explore the intricacies of this technique, highlighting its practical applications and advantages over traditional methods.
Resources and Materials 🍇
- The Image Dataset: Google Drive
- The Software: PostShot and CloudCompare
- The Presentation Slides: My Slide Deck .pdf (Copyrighted; please cite for re-use)
- Additionnal Resources
- 3D Gaussian Splatting for Real-Time Radiance Field Rendering: Project Page (incl. Article, Github)
- State of The Art Survey on 3D Gaussian Splatting: Research Paper (.pdf)
- Awesome 3D Gaussian Splatting Resources: A Repo with Several Resources
3D Gaussian Splatting: The Overall Workflow
The session begins with an introduction to 3D Gaussian Splating, outlining the process of transforming a set of images into a 3D Gaussian Splating experience. The workflow shows several steps as illustrated below:
Note: I emphasize the importance of having a CUDA-enabled graphics card for replicating the session and recommend specific models like the 3090RTX for efficient local testing. The session deliverables include a live recording, data set links, resource curation, and cheat sheets for efficient workflow. The mission involves starting with an image set and creating 3D Gaussian splits, point clouds, and 3D renderings.
Structure From Motion for 3D Gaussian Splatting
Structure from Motion is the first pass in a Gaussian Splatting workflow, and it is what makes the technique more streamlined than its predecessors, particularly for those in the visual effects (VFX) and real-time rendering industries.
At its core, the concept of Structure from Motion (SfM), is key. It involves estimating the 3D structure of a scene along with the motion parameters (pose) of the camera that captured the images. SfM is commonly used in applications linked with 3D Reconstruction Tasks whenever we start from image-based reconstructions. SfM builds on extracted features from multiple images to reconstruct a 3D scene.
This process, from feature extraction and image matching to determining camera poses and sparse point clouds, is the first pass before moving onto the world of Gaussian Splatting. This allows for bypassing the computationally expensive dense matching phase of traditional photogrammetry, offering a faster and more efficient alternative.
3D Gaussian Splatting Fundamentals
Gaussian Splating itself is a fascinating blend of rasterization and optimization.
The technique involves representing 3D data using Gaussians—each defined by a position, a covariance (which describes its stretch and orientation), color, and an alpha channel.
Combining these Gaussians with a differentiable rasterizer allows bi-directional communication between the 3D and 2D representations.
This enables real-time adjustments and optimizations, ensuring the rendered images closely match the ground truth images.
Hands-On Tutorial for 3D Gaussian Splatting
In order to extend the theoretical setup, we dive into a practical aspects of Gaussian Splating using the Post-Shot software.
This means that we have to go through the steps of importing an image set, configuring the training parameters, and launching the training process. Within minutes, we can obtain a rendering of detailed 3D scenes, allowing real-time adjustments and visualizations.
This demonstrates the technique’s accessibility and efficiency, making it a valuable tool for professionals in various fields.
Conclusion
The session provided a comprehensive overview of 3D Gaussian Splating, showcasing its potential to revolutionize 3D reconstruction and rendering. Gaussian Splating offers a faster, more efficient, and highly detailed approach to 3D modeling by blending traditional photogrammetry principles with modern computational techniques.
As technology advances, methods like these will undoubtedly play a crucial role in shaping the future of digital content creation, offering new possibilities for innovation and creativity.
Frequently Asked Questions About 3D Gaussian Splatting
It represents a scene as a set of 3D Gaussians, each defined by a position, a covariance that describes its stretch and orientation, a colour and an alpha channel. A differentiable rasteriser renders those Gaussians and optimises them until the rendered images match the captured ones.
Structure from Motion first estimates the camera poses and a sparse point cloud from your images. The Gaussians are then optimised against the source photographs through a differentiable rasteriser, which lets the 3D and 2D representations correct each other until the renders agree with the input.
Import an image set into PostShot, configure the training parameters, then launch the training. Within minutes you get a rendered 3D scene you can adjust and inspect in real time. CloudCompare is useful for looking at the point cloud that comes out.
Yes. A CUDA-enabled graphics card is what makes replicating this session practical, and a 3090 RTX class card is the one I recommend for testing locally. The training step is the part that needs the GPU.
The original Inria project page hosts both the paper and the GitHub repository, and a state of the art survey is on arXiv. Both are linked in the resources section near the top of this article.
My 3D Recommendation 🍉
Generating 3D Point Clouds from a Set of Images is fantastic. It opens up a lot of horizons… For Creativity purposes only (at this stage). If you seek a more metrically controlled approach, I highly recommend getting expertise with Photogrammetry to unlock Multi-View Reconstruction. It is more constraining than single images but provides a much better geometric base layer. If you want to use it in your solutions, I recommend getting on the 3D Collector’s Pack.
If you want to get a tad more toward application-first or depth-first approaches, I curated several learning tracks and courses on this website to help you along your journey. Feel free to follow the ones that best suit your needs, and do not hesitate to reach out directly to me if you have any questions or if I can advise you on your current challenges!
Open-Access Knowledge
- Medium Tutorials and Articles: Standalone Guides and Tutorials on 3D Data Processing (5′ to 45′)
- Research Papers and Articles: Research Papers published as part of my public R&D work.
- Email Course: Access a 7-day E-Mail Course to Start your 3D Journey
- Youtube Education: Not articles, not research papers, open videos to learn everything around 3D.
3D Online Courses
- 3D Python Crash Course (Complete Standalone). 3D Coding with Python.
- 3D Reconstruction Course (Complete Standalone): Open-Source 3D Reconstruction (incl. NERF and Gaussian Splatting).
- 3D Point Cloud Course (Complete Standalone): Pragmatic, Complete, and Commercial Point Cloud Workflows.
- 3D Object Detection Course (3D Python Recommended): Practical 3D Object Detection with 3D Python.
- 3D Deep Learning Course (3D Python Recommended): Most-Advanced Course for Top AI Engineers and Researchers
3D Learning Tracks
- 3D Segmentation Deck: From Classical 3D Segmentation to 3D Deep Learning and Unsupervised Applications
- 3D Collector’s Pack: Complete Course Track to address both 3D Application and Code Layers.
