1 Presenter: Xu Cao (曹 旭) @ CyberAgent Oct. 9, 2024

What Why How 2 Make the ray-traced 3D Gaussians efficient (~150 FPS) Ray tracing allows rendering secondary effects and distorted cameras that rasterization-based 3DGS cannot, but trades off rendering speed. • An efficient bounding volume hierarchy (BVH) building strategy for 3D Gaussians • An efficient 𝑘-hits based ray-tracing strategy

Background: 3D Gaussian Splatting • Target scene is represented as a set of anisotropic 3D Gaussians with parameters o Center location 𝝁 o Covariance Σ = 𝐑⊤𝐒⊤𝐒𝐑 o Opacity 𝜎 o Spherical harmonics coefficients for view-dependent colors • For rendering, projecting all 3D Gaussians onto the image plane (Rasterization) and perfrom volume rendering 3 Continous form Discrete form

Disadvantages of 3DGS • Rasterization cannont efficiently simulate secondary light transport effects o shadows, relections, and refractions • Rasterization breaks down for highly distorted camera models o Fisheye camera with large FoV o Video that contain rolling shutter effects 4

5 What can we do with ray-traced Gaussians?

Secondary effects: Shadow 6

Secondary effects: Refraction 7

Secondary effects: Reflection 8

Camera distortion: Fisheye simulation 9

Camera distortion: Depth of field simulation 10

Real-time instancing 11

Reconstructing autonomous vehicle scenes 12 Ray-tracing is well-suited for handling distorted camera models and rolling shutter effects simoutanously Left camera Middle camera Right camera

Results from fisheye camera inputs 13

Effiecient forward rendering pipeline 1. Bound each 3DG with primitive geometries and construct a BVH from bounding primitives 2. Query the ray-primitive hits within a sub-inteval on the ray (a chunk) 3. Compute the response of hitted 3DGs within the chunk 4. Perform volume rendering for the chunk of points 15

Effiecient forward rendering pipeline 16

Background: Bounding Volume Hierarchy (BVH) Motivation: For fast ray-primitive intersection query How: • Bound each scene object by a bounding primitive (axis-aligned bounding box in the example) • Construct a tree structure for the bounding primitives • Determine the ray-primitive intersection from root to leaf Benefits: • Determine the intersection between a ray and an AABB is more efficient than, e.g., a triangle mesh • Tree structure reduces intersection query number from 𝑂 𝑛 to 𝑂(log 𝑛), 𝑛 being the number of scene objects Disadvantage: • Require re-building BVH once the scene geometry changes 17

Bounding primitive of a 3D Gaussian Loosely bounded Tightly bounded Trade-off between accuray and efficiency More efficient ray-primitive intersection query More false-positive intersections Icosahedron 二十面体 18

Bounding primitive of a 3D Gaussian Icosahedron 二十面体 1. Initialize the bounding primitive as an icosahedron with a unit inner-sphere 2. Then anisotropically scale the icosahedron vertices based on the 3DG parameters Icosahedron vertices 3DG opacity 3DG center 3DG scaling & rotation Hyperparameter 𝛼𝑚𝑖𝑛 : points with 𝜌 𝒙 > 𝛼𝑚𝑖𝑛 must be included in the icosahedron Including 3DG opacity adaptively adjusts the icosahedron size → Smaller icosahedron for more transparent 3DG (samller 𝜎) → Contribute to faster rendering 19

BVH build time and FPS AABB w/ adaptive clamping w/o adaptive clamping Tighter bounding increases BVH building time (Still acceptable: 0.1sec for 3M Gaussians, only conducted during optimization) But tighter bounding accelerates rendering (Because for each ray, only 3DGs significantly contributing to its color remain) 20

Effiecient forward rendering pipeline 21

𝑘-hits based ray tracing • Cast the ray, and find the first 𝑘 ray-primitive hits o Render the color & transmittance for that batch of hits • Cast the ray again from the last primitive, and find the next 𝑘 hits o Update the color & transmittance using new hits • Repeat until o All primitives are handled o A pre-defined minimal transmittance is reached (because the remaining hitted 3DGs on that ray almost contributes nothing the color) 𝑘 = 3 in this example 22

Why can we split the color rendering? 23 The numerical integration for volume rendering, essentially an ordered sum of weighed colors 𝑖-th intersected Gaussian on a ray, sorted in depth order 𝑳 𝒐, 𝒅 = ෍ 𝑖=1 𝑁 ො 𝒄𝑖 = ෍ 𝑖=1 𝑘 ො 𝒄𝑖 + ෍ 𝑖=𝑘+1 2𝑘 ො 𝒄𝑖 + … + ෍ 𝑖=𝑚𝑘+1 𝑁 ො 𝒄𝑖 The sum can be split into ordered chunks of sums

Why this 𝑘-hits tracing strategy? Authors test different strategies and find the proposed one achieves the best balance between rendering quality and efficiency Tested different tracing strategies 24

Effiecient forward rendering pipeline 25

Evaluating particle response • Evaluating the integral requires samples from intersected 3DG • Each 3DG contributes one sample point to volume rendering • The sample point is the max response on the ray • And can be analytically computed as • Orthogonally projecting the 3DG center onto the ray causes errors for stretched anisotropic 3DGs 𝜏𝑚𝑎𝑥 = argmax𝜏 𝜌(𝒐 + 𝜏𝒅) 𝜏𝑚𝑎𝑥 = 𝝁−𝒐 Σ−1𝒅 𝒅⊤𝚺−𝟏𝒅 𝒐 𝒅 26

Effiecient forward rendering pipeline 27

Backward pass for optimization • Auto-differentiation (e.g., PyTorch) is not available • Run the forward pass first to evaluate the loss functions • Then re-cast the same rays and sample the same 3DGs to compute the derivatives and update the 3DG parameters (Implemented by hand) • Optimization strategy follows the standard 3DGS • BVH is rebuilt every iteration (in total 30k iters * 0.1 sec ≈ 50 min) [Kerbl+, 3DGS] Cloning and Splitting 28

Proposal generalizes to different kernal functions a.k.a. 2DGS 𝑛 = 2 in this paper 29

Effect of different kernal functions Close rendering quality Twice faster 30

Faster rendering if #hits per ray is smaller Each pixel indicates the #hitted primitives per ray Generalized Gaussian Gaussian A generalized Gaussian is denser/smaller than an ordinary Gaussian, thus reducing the #hits per ray 31

Comparison against baselines • Baselines • 3DGS, MipNeRF360, Instant-NGP, Plenoxels • Benchmarks • Tanks & temples, MipNeRF360, DeepBlending, NeRFSynthetic • Variants of the proposal • Ours (reference): as close setting as 3DGS • Ours: hyperparameters and strategies are finetuned based on ablations 32

Comparison against baselines 33

Comparison against baselines 34

Comparison against baselines 35

Ablation: Maximum number of Gaussians 36 This work heuristically sets the max number of Gaussians in a scene as 3M, because they find further increasing the number does not improve rendering quality

Summary • An efficient ray-tracing pipeline for semi-transparent particles o Ray tracing particles are not uncommon (e.g., Fuzzy metaballs*) o BVH largely reduce the ray-tracing time ❖We do not need to evaluate all Gaussians to render one ray ❖Only ray-primitive intersected Gaussians are considered for rendering • Ray tracing eases rendering secondary light transport effects and handle or simulate distorted camera models • Relighting & inverse rendering are left as future works 37 * Keselman and Hebert, Approximate Differentiable Rendering with Algebraic Surfaces, ECCV’22