3D Gaussian Splatting for Real-Time Radiance Field Rendering

Reading guide

Problem and motivation

Neural Radiance Fields (NeRFs) have transformed novel view synthesis by optimizing continuous implicit scene representations via Multi-Layer Perceptrons (MLPs). However, rendering these continuous fields relies on stochastic volumetric ray-marching, which requires evaluating tens or hundreds of samples per ray. This leads to heavy computational burdens, long training times (e.g., up to 48 hours for Mip-NeRF360), and very slow inference speeds (seconds per frame), preventing real-time navigation at high resolutions like 1080p.

While faster radiance field variants (such as Plenoxels and InstantNGP) rely on voxel or hash grids to accelerate training, they often trade off visual fidelity and struggle with empty space or high-frequency detail. Conversely, point-based approaches offer explicit geometry that is GPU-friendly, but traditional point rasterization requires dense Multi-View Stereo (MVS) geometry to avoid holes, or relies on CNN post-processing that introduces temporal flickering.

This paper proposes an unstructured, explicit scene representation using 3D Gaussians combined with a fast tile-based rasterizer, achieving the rendering fidelity of top-tier implicit radiance fields alongside real-time frame rates and competitive optimization times.

Key ideas

1. Unstructured 3D Gaussian Representation: Instead of continuous coordinate networks or dense grids, the radiance field is parameterized by discrete 3D Gaussians with full anisotropic covariance matrices, initialized from sparse Structure-from-Motion (SfM) point clouds without requiring normals or dense MVS.
2. Parameterization via Scale and Rotation: To ensure the covariance matrix remains positive semi-definite throughout gradient descent, it is decoupled into a 3D scaling vector and a unit quaternion.
3. Adaptive Density Control: The optimization dynamically manages scene density by tracking view-space position gradients: cloning under-reconstructed Gaussians and splitting over-reconstructed Gaussians, while pruning transparent or overly large splats.
4. Visibility-Aware Tile-Based Rasterization: A custom GPU rasterizer organizes screen pixels into $16\times 16$ tiles, depth-sorts splats per frame using a fast Radix sort, and alpha-blends anisotropic projections front-to-back without limiting the number of splats that receive gradient updates.

Method

1. 3D Gaussian Scene Representation

A 3D Gaussian centered at mean $\mu$ is defined by its covariance matrix $\Sigma$ in world space: $$G(x)=e^{-\frac{1}{2}(x-\mu {)}^T{\Sigma }^{-1}(x-\mu )}$$

To guarantee that $\Sigma$ remains positive semi-definite during optimization, it is factorized into a rotation matrix $R$ and a scaling matrix $S$: $$\Sigma =RS{S}^T{R}^T$$ Here, $S$ is stored as a 3D vector $s$ evaluated with an exponential activation, and $R$ is parameterized by a normalized unit quaternion $q$.

For rendering, 3D Gaussians are projected to 2D image coordinates following the EWA splatting framework. Given a viewing transformation $W$ and the Jacobian $J$ of the projective transformation's affine approximation, the 2D covariance matrix ${\Sigma }^{\prime }$ in screen space is: $${\Sigma }^{\prime }=JW\Sigma W^T{J}^T$$ The third row and column are discarded, yielding a $2\times 2$ covariance matrix for planar splatting.

2. Rendering and Image Formation

Each Gaussian has an opacity $\alpha$ (constrained to $[0,1)$ via a sigmoid activation) and spherical harmonics (SH) coefficients describing view-dependent color $\mathbf{c}$. Blending $N$ ordered overlapping Gaussians at a pixel follows the standard volumetric formulation: $$C=\sum _{i\in \mathcal{N}}{\mathbf{c}}_i{\alpha }_i\prod _{j=1}^{i-1}(1-{\alpha }_j)$$ where ${\alpha }_i$ is determined by evaluating the 2D projected Gaussian multiplied by the learned per-point opacity.

3. Optimization and Adaptive Control

The loss function combines an ${\mathcal{L}}_1$ loss with a D-SSIM loss (with $\lambda =0.2$): $$\mathcal{L}=(1-\lambda ){\mathcal{L}}_1+\lambda {\mathcal{L}}_{\text{D-SSIM}}$$

Adaptive density control runs every 100 iterations on Gaussians whose average view-space positional gradient exceeds ${\tau }_{\text{pos}}=0.0002$:

• Clone (Under-reconstruction): For small Gaussians, a duplicate is created and moved along the positional gradient direction.
• Split (Over-reconstruction): For large Gaussians, the primitive is replaced by two smaller Gaussians whose scales are reduced by a factor of $\varphi =1.6$, sampling their initial positions from the parent Gaussian.
• Pruning: Prunes Gaussians where $\alpha <{ϵ}_{\alpha }$ or those excessively large in world/view space. Every $N=3000$ iterations, opacities are reset near zero to filter out floating artifacts.

4. Fast Differentiable Tile-Based Rasterizer

• The screen is divided into $16\times 16$ pixel tiles.
• Gaussians passing 99% confidence interval frustum culling are assigned 64-bit keys encoding tile ID and view-space depth.
• A single GPU Radix sort orders all instances across the frame.
• Thread blocks accumulate color per tile front-to-back until $\alpha$ reaches saturation (stopping before exceeding $0.9999$).
• The backward pass traverses tiles back-to-front, recovering intermediate opacities by dividing the final accumulated opacity by each primitive's $\alpha$, avoiding expensive dynamic per-pixel memory buffers.

Results

• Mip-NeRF360 Dataset: After 30K iterations (41m 33s average training time), 3D Gaussian Splatting achieves an average SSIM of 0.815, PSNR of 27.21 dB, and LPIPS of 0.214, matching or exceeding Mip-NeRF360 (SSIM 0.792, PSNR 27.69 dB, LPIPS 0.237, 48 hours training), while rendering at 134 FPS compared to Mip-NeRF360's 0.06 FPS.
• Fast Configuration (7K iterations): Delivers an SSIM of 0.770 and PSNR of 25.60 dB in just 6m 25s at 160 FPS, outperforming InstantNGP-Base (SSIM 0.671, 11.7 FPS) and Plenoxels (SSIM 0.626, 6.79 FPS) within comparable training times.
• Tanks&Temples & Deep Blending: On Tanks&Temples, 30K iterations reach an SSIM of 0.841 and PSNR of 23.14 dB at 154 FPS. On Deep Blending, it achieves an SSIM of 0.903 and PSNR of 29.41 dB at 137 FPS.
• Synthetic Blender: Scores an average PSNR of 33.32 dB from a completely random initialization of 100K points, rendering at 180–300 FPS.

Limitations and open questions

• Unobserved Views and Outliers: Sparse coverage or views far from training trajectories can produce visible artifacts, such as elongated or "splotchy" Gaussians.
• Popping Artifacts: Sudden switching in depth ordering or aggressive guard-band culling can introduce popping during navigation, pointing to the need for better antialiasing.
• Memory Consumption: Although compact compared to classic dense point clouds, the explicit representation requires hundreds of megabytes (e.g., 400–750 MB for scenes with 1–5 million Gaussians), and unoptimized prototype training can spike GPU memory above 20 GB.
• Lack of Regularization: The framework does not include explicit geometric smoothness or density regularizers, leaving room for background floaters in regions poorly constrained by camera rays.