NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis

Ben Mildenhall; Pratul P. Srinivasan; Matthew Tancik; Jonathan T. Barron; Ravi Ramamoorthi; Ren Ng

Papers · arXiv 2003.08934

NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis

Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, Ren Ng

Published 2020-03-19cs.CVcs.GR arXiv PDF

Read with translation and AI

Summary in three lines

🎯 Synthesizing high-resolution photorealistic novel views of complex 3D scenes without discrete voxel grids or proxy mesh geometry.
🔧 Representing static scenes as continuous 5D functions mapped by an MLP using positional encoding, differentiable volume rendering, and hierarchical sampling.
📈 Achieved state-of-the-art view synthesis, reaching 31.01 dB PSNR on realistic synthetic scenes and compressing scene representations down to ~5 MB.

Abstract

We present a method that achieves state-of-the-art results for synthesizing novel views of complex scenes by optimizing an underlying continuous volumetric scene function using a sparse set of input views. Our algorithm represents a scene using a fully-connected (non-convolutional) deep network, whose input is a single continuous 5D coordinate (spatial location $(x, y, z)$ and viewing direction $(θ, φ)$ ) and whose output is the volume density and view-dependent emitted radiance at that spatial location. We synthesize views by querying 5D coordinates along camera rays and use classic volume rendering techniques to project the output colors and densities into an image. Because volume rendering is naturally differentiable, the only input required to optimize our representation is a set of images with known camera poses. We describe how to effectively optimize neural radiance fields to render photorealistic novel views of scenes with complicated geometry and appearance, and demonstrate results that outperform prior work on neural rendering and view synthesis. View synthesis results are best viewed as videos, so we urge readers to view our supplementary video for convincing comparisons.

Reading guide

Problem and motivation

Novel view synthesis from captured imagery has traditionally relied on discrete volumetric representations (such as voxel grids or multiplane images) or surface meshes. However, discretized voxel grids incur prohibitive memory and compute footprints that scale cubically with resolution, while mesh-based differentiable optimization suffers from poor optimization landscapes and requires good initial topology. Prior implicit neural scene representations mapped coordinates directly to signed distance functions or single-surface points, struggling with complex geometry, view-dependent effects (like specularities), and high-frequency details, yielding oversmoothed results.

Key ideas

Continuous 5D Scene Function: Represent a static scene as an implicit continuous function $F_{Θ} : (x, d) \to (c, σ)$ , mapping a 3D location $x = (x, y, z)$ and 2D viewing direction $d = (θ, ϕ)$ to emitted RGB radiance $c$ and volume density $σ$ .
Multiview Consistency via Asymmetric Prediction: Density $σ$ is predicted solely from position $x$ (geometry is view-invariant), whereas emitted color $c$ depends on both position $x$ and viewing direction $d$ (enabling specular and non-Lambertian highlights).
High-Frequency Positional Encoding: Overcoming the spectral bias of MLPs towards low frequencies by mapping low-dimensional input coordinates to high-dimensional spaces using sinusoidal basis functions.
Differentiable Volume Rendering & Hierarchical Sampling: Employing numerical quadrature to accumulate continuous samples along rays into 2D pixel colors, optimized end-to-end against input photos, accelerated by coarse-to-fine hierarchical sampling.

Method (with the key equations)

For a camera ray $r (t) = o + t d$ bounded by near and far planes $t_{n}$ and $t_{f}$ , the continuous expected color $C (r)$ is: $C (r) = \int_{t_{n}}^{t_{f}} T (t) σ (r (t)) c (r (t), d) d t, T (t) = exp (- \int_{t_{n}}^{t} σ (r (s)) d s)$ where $T (t)$ is accumulated transmittance.

To compute this numerically, the continuous interval is partitioned into $N$ bins and evaluated with stratified sampling, yielding discrete quadrature equivalent to alpha compositing: $\hat{C} (r) = i = 1 \sum N T_{i} (1 - exp (- σ_{i} δ_{i})) c_{i}, T_{i} = exp (- j = 1 \sum i - 1 σ_{j} δ_{j})$ where $δ_{i} = t_{i + 1} - t_{i}$ .

To capture high-frequency geometry and texture, inputs are mapped via positional encoding $γ (p)$ : $γ (p) = (sin (2^{0} π p), cos (2^{0} π p), \dots, sin (2^{L - 1} π p), cos (2^{L - 1} π p))$ with $L = 10$ for $x$ and $L = 4$ for $d$ .

To sample efficiently, NeRF concurrently optimizes two networks (coarse with $N_{c} = 64$ samples and fine with $N_{f} = 128$ additional samples). Coarse sample weights $w_{i} = T_{i} (1 - exp (- σ_{i} δ_{i}))$ define a piecewise-constant probability density function along the ray, from which the fine samples are drawn via inverse transform sampling.

The entire model is trained by minimizing the total squared error over a ray batch $R$ : $L = r \in R \sum [\hat{C}_{c} (r) - C (r)_{2}^{2} + \hat{C}_{f} (r) - C (r)_{2}^{2}]$

Results (with the most important numbers)

Realistic Synthetic $36 0^{\circ}$ : NeRF achieved 31.01 dB PSNR, 0.947 SSIM, and 0.081 LPIPS, substantially outperforming baselines such as NV (26.05 dB), LLFF (24.88 dB), and SRN (22.26 dB).
Diffuse Synthetic $36 0^{\circ}$ : NeRF obtained 40.15 dB PSNR and 0.991 SSIM, compared to DeepVoxels (30.55 dB mean implied by per-scene scores) and LLFF (34.38 dB).
Real Forward-Facing Scenes: Achieved 26.50 dB PSNR, surpassing LLFF (24.13 dB) and SRN (22.84 dB).
Storage Efficiency: A fully trained NeRF model requires approximately 5 MB in MLP weights—about a $3000 \times$ storage reduction compared to LLFF voxel representations (>15 GB per scene).

Limitations and open questions

Training and Rendering Time: Optimizing a scene takes 100k–300k iterations (roughly 1–2 days on a single NVIDIA V100 GPU). Rendering a single frame requires 150–200 million MLP queries, taking ~30 seconds.
Model Interpretability: Unlike voxel grids or polygon meshes, the scene is locked inside the weights of an MLP, making direct editing, inspection, and error debugging difficult.
Controlled Capture Requirements: The method assumes known camera poses (e.g., from COLMAP), static scenes, consistent lighting, and clean scene bounding volumes.

Key terms

Neural Radiance Field (NeRF): A continuous 5D volumetric scene representation parameterized by a deep fully-connected neural network that maps spatial coordinates and viewing directions to volume density and emitted radiance.
Positional Encoding: A deterministic sinusoidal mapping $γ (p)$ that projects continuous input coordinates into higher-dimensional frequency spaces, enabling the network to learn high-frequency details.
Volume Density: A view-independent differential opacity $σ (x)$ that models the probability of a camera ray terminating at an infinitesimal point in 3D space.
Transmittance: The probability $T (t)$ that a camera ray travels from the near clipping plane to a specific distance along the ray without terminating on any scene particles.
Hierarchical Volume Sampling: A coarse-to-fine sampling method that uses ray densities estimated by a coarse network to construct a non-uniform probability distribution for sampling additional points along the ray for a fine network.
Stratified Sampling: A sampling procedure that splits the ray's depth range into evenly-spaced bins and uniformly draws one random continuous sample per bin to enable continuous evaluation during optimization.
Normalized Device Coordinates (NDC): A coordinate space mapping the perspective viewing frustum of forward-facing captures to $[- 1, 1]^{3}$ , making the camera axis linear in disparity.

Questions

Why is density conditioned only on position, while color is conditioned on position and viewing direction?

Restricting density $σ$ to depend solely on spatial location $x$ enforces multiview geometric consistency across viewpoints, while allowing color $c$ to take viewing direction $d$ as input enables the MLP to represent view-dependent non-Lambertian phenomena, such as specular reflections.

How does positional encoding in NeRF differ from positional encoding in Transformers?

Transformers use positional encodings to inject discrete token order into permutation-invariant attention blocks, whereas NeRF uses sinusoidal encodings to map continuous low-dimensional coordinates into a higher-dimensional space to bypass the spectral bias of MLPs toward low frequencies.

What inputs are needed to optimize a NeRF from scratch?

NeRF requires only a set of calibrated 2D RGB images of a static scene with known camera intrinsics, camera poses, and near/far scene bounds (derived using COLMAP for real data or ground-truth poses for synthetic data).

How long does NeRF take to train and render?

Training a single scene takes between 100k and 300k iterations (roughly 1 to 2 days) on an NVIDIA V100 GPU. Rendering novel views requires evaluating 150 to 200 million coordinate queries per image, taking approximately 30 seconds per frame.

The overview on this page was written by AI from the paper and may contain mistakes. Read the paper for the details.

Read with translation and AI