[論文読み] 3D Gaussian Ray Tracing: Fast Tracing of Particle Scenes

- 3D Gaussianで表現されたシーンを効率的にレイトレシングする手法の提案
- レンダリング速度は従来の3DGSの1/2 ~ 1/3になるが、リアルタイムで影、鏡面反射、屈折といった光の伝播が表現可能　
- 各3D Gaussianを二十面体で囲み、BVHを構築することで、レイトレシングのコストを大幅に削減

Transcript

1. 1 Presenter: Xu Cao (曹 旭) @ CyberAgent Oct. 9, 2024
2. What Why How 2 Make the ray-traced 3D Gaussians efficient (~150 FPS) Ray tracing allows rendering secondary effects and distorted cameras that rasterization-based 3DGS cannot, but trades off rendering speed. • An efficient bounding volume hierarchy (BVH) building strategy for 3D Gaussians • An efficient 𝑘-hits based ray-tracing strategy
3. Background: 3D Gaussian Splatting • Target scene is represented as a set of anisotropic 3D Gaussians with parameters o Center location 𝝁 o Covariance Σ = 𝐑⊤𝐒⊤𝐒𝐑 o Opacity 𝜎 o Spherical harmonics coefficients for view-dependent colors • For rendering, projecting all 3D Gaussians onto the image plane (Rasterization) and perfrom volume rendering 3 Continous form Discrete form
4. Disadvantages of 3DGS • Rasterization cannont efficiently simulate secondary light transport effects o shadows, relections, and refractions • Rasterization breaks down for highly distorted camera models o Fisheye camera with large FoV o Video that contain rolling shutter effects 4
5. 5 What can we do with ray-traced Gaussians?
6. Secondary effects: Shadow 6
7. Secondary effects: Refraction 7
8. Secondary effects: Reflection 8
9. Camera distortion: Fisheye simulation 9
10. Camera distortion: Depth of field simulation 10
11. Real-time instancing 11
12. Reconstructing autonomous vehicle scenes 12 Ray-tracing is well-suited for handling distorted camera models and rolling shutter effects simoutanously Left camera Middle camera Right camera
13. Results from fisheye camera inputs 13
14. Effiecient forward rendering pipeline 1. Bound each 3DG with primitive geometries and construct a BVH from bounding primitives 2. Query the ray-primitive hits within a sub-inteval on the ray (a chunk) 3. Compute the response of hitted 3DGs within the chunk 4. Perform volume rendering for the chunk of points 15
15. Effiecient forward rendering pipeline 16
16. Background: Bounding Volume Hierarchy (BVH) Motivation: For fast ray-primitive intersection query How: • Bound each scene object by a bounding primitive (axis-aligned bounding box in the example) • Construct a tree structure for the bounding primitives • Determine the ray-primitive intersection from root to leaf Benefits: • Determine the intersection between a ray and an AABB is more efficient than, e.g., a triangle mesh • Tree structure reduces intersection query number from 𝑂 𝑛 to 𝑂(log 𝑛), 𝑛 being the number of scene objects Disadvantage: • Require re-building BVH once the scene geometry changes 17
17. Bounding primitive of a 3D Gaussian Loosely bounded Tightly bounded Trade-off between accuray and efficiency More efficient ray-primitive intersection query More false-positive intersections Icosahedron 二十面体 18
18. Bounding primitive of a 3D Gaussian Icosahedron 二十面体 1. Initialize the bounding primitive as an icosahedron with a unit inner-sphere 2. Then anisotropically scale the icosahedron vertices based on the 3DG parameters Icosahedron vertices 3DG opacity 3DG center 3DG scaling & rotation Hyperparameter 𝛼𝑚𝑖𝑛 : points with 𝜌 𝒙 > 𝛼𝑚𝑖𝑛 must be included in the icosahedron Including 3DG opacity adaptively adjusts the icosahedron size → Smaller icosahedron for more transparent 3DG (samller 𝜎) → Contribute to faster rendering 19
19. BVH build time and FPS AABB w/ adaptive clamping w/o adaptive clamping Tighter bounding increases BVH building time (Still acceptable: 0.1sec for 3M Gaussians, only conducted during optimization) But tighter bounding accelerates rendering (Because for each ray, only 3DGs significantly contributing to its color remain) 20
20. Effiecient forward rendering pipeline 21
21. 𝑘-hits based ray tracing • Cast the ray, and find the first 𝑘 ray-primitive hits o Render the color & transmittance for that batch of hits • Cast the ray again from the last primitive, and find the next 𝑘 hits o Update the color & transmittance using new hits • Repeat until o All primitives are handled o A pre-defined minimal transmittance is reached (because the remaining hitted 3DGs on that ray almost contributes nothing the color) 𝑘 = 3 in this example 22
22. Why can we split the color rendering? 23 The numerical integration for volume rendering, essentially an ordered sum of weighed colors 𝑖-th intersected Gaussian on a ray, sorted in depth order 𝑳 𝒐, 𝒅 = ෍ 𝑖=1 𝑁 ො 𝒄𝑖 = ෍ 𝑖=1 𝑘 ො 𝒄𝑖 + ෍ 𝑖=𝑘+1 2𝑘 ො 𝒄𝑖 + … + ෍ 𝑖=𝑚𝑘+1 𝑁 ො 𝒄𝑖 The sum can be split into ordered chunks of sums
23. Why this 𝑘-hits tracing strategy? Authors test different strategies and find the proposed one achieves the best balance between rendering quality and efficiency Tested different tracing strategies 24
24. Effiecient forward rendering pipeline 25
25. Evaluating particle response • Evaluating the integral requires samples from intersected 3DG • Each 3DG contributes one sample point to volume rendering • The sample point is the max response on the ray • And can be analytically computed as • Orthogonally projecting the 3DG center onto the ray causes errors for stretched anisotropic 3DGs 𝜏𝑚𝑎𝑥 = argmax𝜏 𝜌(𝒐 + 𝜏𝒅) 𝜏𝑚𝑎𝑥 = 𝝁−𝒐 Σ−1𝒅 𝒅⊤𝚺−𝟏𝒅 𝒐 𝒅 26
26. Effiecient forward rendering pipeline 27
27. Backward pass for optimization • Auto-differentiation (e.g., PyTorch) is not available • Run the forward pass first to evaluate the loss functions • Then re-cast the same rays and sample the same 3DGs to compute the derivatives and update the 3DG parameters (Implemented by hand) • Optimization strategy follows the standard 3DGS • BVH is rebuilt every iteration (in total 30k iters * 0.1 sec ≈ 50 min) [Kerbl+, 3DGS] Cloning and Splitting 28
28. Proposal generalizes to different kernal functions a.k.a. 2DGS 𝑛 = 2 in this paper 29
29. Effect of different kernal functions Close rendering quality Twice faster 30
30. Faster rendering if #hits per ray is smaller Each pixel indicates the #hitted primitives per ray Generalized Gaussian Gaussian A generalized Gaussian is denser/smaller than an ordinary Gaussian, thus reducing the #hits per ray 31
31. Comparison against baselines • Baselines • 3DGS, MipNeRF360, Instant-NGP, Plenoxels • Benchmarks • Tanks & temples, MipNeRF360, DeepBlending, NeRFSynthetic • Variants of the proposal • Ours (reference): as close setting as 3DGS • Ours: hyperparameters and strategies are finetuned based on ablations 32
32. Comparison against baselines 33
33. Comparison against baselines 34
34. Comparison against baselines 35
35. Ablation: Maximum number of Gaussians 36 This work heuristically sets the max number of Gaussians in a scene as 3M, because they find further increasing the number does not improve rendering quality
36. Summary • An efficient ray-tracing pipeline for semi-transparent particles o Ray tracing particles are not uncommon (e.g., Fuzzy metaballs*) o BVH largely reduce the ray-tracing time ❖We do not need to evaluate all Gaussians to render one ray ❖Only ray-primitive intersected Gaussians are considered for rendering • Ray tracing eases rendering secondary light transport effects and handle or simulate distorted camera models • Relighting & inverse rendering are left as future works 37 * Keselman and Hebert, Approximate Differentiable Rendering with Algebraic Surfaces, ECCV’22