This Tutorial targets 3D Reconstruction and 3D Real-Time Visualization leveraging 3D Gaussian Splatting. In the end, you will be able to generate 3D Point Clouds and 3D Renderings from a Set of Images and/or Videos.
3D Reconstruction with 3D Gaussian Splatting
In the rapidly evolving landscape of 3D modeling, innovations consistently push the envelope, enhancing our capabilities to create, visualize, and interact with digital objects.
One such groundbreaking technique that’s making waves is 3D Gaussian Splatting. Though rooted in decades-old principles, this method leverages modern computational power to transform how we handle and render 3D data from simple 2D images or videos.
In this Live Course Session, we explore the intricacies of this technique, highlighting its practical applications and advantages over traditional methods.
Resources and Materials 🍇
- The Image Dataset: Google Drive
- The Software: PostShot and CloudCompare
- The Presentation Slides: My Slide Deck .pdf (Copyrighted; please cite for re-use)
- Additionnal Resources
- 3D Gaussian Splatting for Real-Time Radiance Field Rendering: Project Page (incl. Article, Github)
- State of The Art Survey on 3D Gaussian Splatting: Research Paper (.pdf)
- Awesome 3D Gaussian Splatting Resources: A Repo with Several Resources
3D Gaussian Splatting: The Overall Workflow
The session begins with an introduction to 3D Gaussian Splating, outlining the process of transforming a set of images into a 3D Gaussian Splating experience. The workflow shows several steps as illustrated below:
Note: I emphasize the importance of having a CUDA-enabled graphics card for replicating the session and recommend specific models like the 3090RTX for efficient local testing. The session deliverables include a live recording, data set links, resource curation, and cheat sheets for efficient workflow. The mission involves starting with an image set and creating 3D Gaussian splits, point clouds, and 3D renderings.
Structure From Motion for 3D Gaussian Splatting
This historical context underscored the field’s continuous evolution, leading to the current focus on Gaussian Splatting. Unlike its predecessors, this technique promises a more streamlined and efficient workflow, particularly beneficial for those in the visual effects (VFX) and real-time rendering industries.
At its core, the concept of Structure from Motion (SfM), is key. It involves estimating the 3D structure of a scene along with the motion parameters (pose) of the camera that captured the images. SfM is commonly used in applications linked with 3D Reconstruction Tasks whenever we start from image-based reconstructions. SfM builds on extracted features from multiple images to reconstruct a 3D scene.
This process, from feature extraction and image matching to determining camera poses and sparse point clouds, is the first pass before moving onto the world of Gaussian Splatting. This allows for bypassing the computationally expensive dense matching phase of traditional photogrammetry, offering a faster and more efficient alternative.
3D Gaussian Splatting Fundamentals
Gaussian Splating itself is a fascinating blend of rasterization and optimization.
The technique involves representing 3D data using Gaussians—each defined by a position, a covariance (which describes its stretch and orientation), color, and an alpha channel.
Combining these Gaussians with a differentiable rasterizer allows bi-directional communication between the 3D and 2D representations.
This enables real-time adjustments and optimizations, ensuring the rendered images closely match the ground truth images.
Hands-On Tutorial for 3D Gaussian Splatting
In order to extend the theoretical setup, we dive into a practical aspects of Gaussian Splating using the Post-Shot software.
This means that we have to go through the steps of importing an image set, configuring the training parameters, and launching the training process. Within minutes, we can obtain a rendering of detailed 3D scenes, allowing real-time adjustments and visualizations.
This demonstrates the technique’s accessibility and efficiency, making it a valuable tool for professionals in various fields.
Conclusion
The session provided a comprehensive overview of 3D Gaussian Splating, showcasing its potential to revolutionize 3D reconstruction and rendering. Gaussian Splating offers a faster, more efficient, and highly detailed approach to 3D modeling by blending traditional photogrammetry principles with modern computational techniques.
As technology advances, methods like these will undoubtedly play a crucial role in shaping the future of digital content creation, offering new possibilities for innovation and creativity.
My 3D Recommendation 🍉
Generating 3D Point Clouds from a Set of Images is fantastic. It opens up a lot of horizons… For Creativity purposes only (at this stage). If you seek a more metrically controlled approach, I highly recommend getting expertise with Photogrammetry to unlock Multi-View Reconstruction. It is more constraining than single images but provides a much better geometric base layer. If you want to use it in your solutions, I recommend getting on the 3D Collector’s Pack.
If you want to get a tad more toward application-first or depth-first approaches, I curated several learning tracks and courses on this website to help you along your journey. Feel free to follow the ones that best suit your needs, and do not hesitate to reach out directly to me if you have any questions or if I can advise you on your current challenges!
Open-Access Knowledge
- Medium Tutorials and Articles: Standalone Guides and Tutorials on 3D Data Processing (5′ to 45′)
- Research Papers and Articles: Research Papers published as part of my public R&D work.
- Email Course: Access a 7-day E-Mail Course to Start your 3D Journey
- Youtube Education: Not articles, not research papers, open videos to learn everything around 3D.
3D Online Courses
- 3D Python Crash Course (Complete Standalone). 3D Coding with Python.
- 3D Reconstruction Course (Complete Standalone): Open-Source 3D Reconstruction (incl. NERF and Gaussian Splatting).
- 3D Point Cloud Course (Complete Standalone): Pragmatic, Complete, and Commercial Point Cloud Workflows.
- 3D Object Detection Course (3D Python Recommended): Practical 3D Object Detection with 3D Python.
- 3D Deep Learning Course (3D Python Recommended): Most-Advanced Course for Top AI Engineers and Researchers
3D Learning Tracks
- 3D Segmentation Deck: From Classical 3D Segmentation to 3D Deep Learning and Unsupervised Applications
- 3D Collector’s Pack: Complete Course Track to address both 3D Application and Code Layers.