Event-based Camera Simulation using Monte Carlo Path Tracing with Adaptive Denoising Yuta Tsuji (Waseda University) Tatsuya Yatagawa (Hitotsubashi University) Hiroyuki Kubo (Chiba University) Shigeo Morishima (Waseda University) Rendering Events

Unlike ordinary RGB camera..., l It captures changes in brightness (see video below) l Shutter can be fired asynchronously on each pixel (high-speed and save battery) l High dynamic range, as high as 140dB (cf., RGB = 60dB) What’s the event-based camera? Sony, Inc. | Event-based Vision Sensor (EVS) to detect only changes in moving subjects: https://www.youtube.com/watch?v=6xOmo7Ikwzk Ordinary RGB camera Event-based camera #1269, MA.PB: Denoising, 9th Oct., 11:00 - 12:30 2

Applications of event-based cameras Ultimate SLAM (SLAM with UAV) [Vidal et al., IEEE RAL 2018] Tracing for fast moving objects [Mitrokhin et al., IROS 2018] High framerate Edge detection & Save power #1269, MA.PB: Denoising, 9th Oct., 11:00 - 12:30 3

Though there are many important applications..., l No event-based camera simulator based on Monte Carlo path tracing (though there’re some image-based simulator) l We cannot obtain physically accurate event videos from 3D synthetic scenes #1269, MA.PB: Denoising, 9th Oct., 11:00 - 12:30 Challenges Event video 3D scene RGB video Image-based simulator MC path tracing Possible approach with conventional techniques 4

Unfortunately, NOT, because, l Image-based simulator requires clean video frames l However, getting clean frames with path tracing (PT) is extremely time-consuming Does conventional technique work? Input video (obtained by path PT) Output from image-based simulator (ESIM) #1269, MA.PB: Denoising, 9th Oct., 11:00 - 12:30 5

With noisy video frames of path tracing, get a clean event video #1269, MA.PB: Denoising, 9th Oct., 11:00 - 12:30 6 Goal & Contributions Our method Contributions: l Apply weighted linear regression (WLR) denoising for robust event detection l Derive a threshold for the residual of WLR to detect events Input video (obtained by path PT)

Denoising with weighted local regression (WLR) Pixel attributes (G-buffer) Color (3D) Position (3D) Normal (3D) (In our method, it’s 9D) #1269, MA.PB: Denoising, 9th Oct., 11:00 - 12:30 7 Input Denoised Question: Does using denoising before image-based event simulation suffice? Originally proposed by [Moon et al., ACM TOG, 2014]

Does denoising work for event detection? ESIM + WLR Input WLR denoised #1269, MA.PB: Denoising, 9th Oct., 11:00 - 12:30 8 As these videos indicate, the straightforward combination of denoising and image-based event simulation does not work

Threshold the residual of WLR, rather than thresholding brightness changes #1269, MA.PB: Denoising, 9th Oct., 11:00 - 12:30 9 Our solution Time Residue Time Brightness The thunder mark “ ” represents the frame where event occurs Residue threshold: 𝛿 Brightness threshold: 𝜏 Prior approach Our approach

Residual thresholding #1269, MA.PB: Denoising, 9th Oct., 11:00 - 12:30 10 WLR residual [Moon et al., ACM TOG 2014] Refer to our paper for the derivation for the relationship: 𝛿 = 𝜏" 𝑅 𝑥; 𝛼, 𝛃 = ) !∈#(%) 𝑤! 𝛼 − 𝛃' 𝐜% − 𝐜! ( 𝐜! : pixel attribute at 𝑥 𝛼, 𝛃 : WLR model parameters 𝑤" : weight for pixel 𝑝 𝑁(𝑥) : neighboring pixels of 𝑥 What we do is: l Solve WLR at time 𝑡 to obtain 𝛼) and 𝛃) l At another frame 𝑠, only evaluate 𝑅(𝑥; 𝛼), 𝛃)) for pixels 𝑥 of 𝑠 l Threshold 𝑅(𝑥; 𝛼), 𝛃)) with 𝛿 to detect events

Why is residue thresholding good? l Our method solves WLR only at the frame where a new event occurs l At other frames, we only calculate the residual of the WLR model #1269, MA.PB: Denoising, 9th Oct., 11:00 - 12:30 11 Time Residual Time Brightness Residual threshold: 𝛿 Brightness threshold: 𝜏 Prior approach Our approach Where we need to solve WLR → So, our method solves WLR only at much fewer frames

For several 3D scenes, we compare the event videos of methods l Input: Noisy input frames obtained by 32-spp MC path tracing l Reference: Clean frames obtained by 4096-spp MC path tracing (we assume events obtained by this is sufficiently accurate) Experiment Scene #1: San Miguel (4096spp) n 4096-spp reference → 18 hours Timing* n 32-spp input → 18 min Scene #2: Living room (4096spp) n 4096-spp reference → 15 hours Timing* n 32-spp input → 9 min * 200✕200 pixels, 240 frames on computed with 3.6 GHz Intel i9-9900K (8 cores) #1269, MA.PB: Denoising, 9th Oct., 11:00 - 12:30 12

WLR+ESIM ESIM Ours Reference (Event, 4096 spp) ESIM WLR+ESIM Ours Time (sec) 3 275 194 F-score 0.789 0.914 0.916 Chamfer dist.(CD) 0.00060 0.00024 0.00021 Refer to our paper for definitions of F- score, and CD. Reference (RGB, 4096 spp) Scene #1: San Miguel ESIM Qualitatively and quantitatively, our method performs the best #1269, MA.PB: Denoising, 9th Oct., 11:00 - 12:30 13

WLR+ESIM ESIM Ours Reference (Event, 4096 spp) ESIM WLR+ESIM Ours Time (sec) 3 231 135 F-score 0.450 0.712 0.739 Chamfer dist.(CD) 0.029 0.0072 0.00089 Refer to our paper for definitions of F- score, and CD. Reference (RGB, 4096 spp) Scene #2: Living room ESIM Again, our method performs the best qualitatively and quantitatively #1269, MA.PB: Denoising, 9th Oct., 11:00 - 12:30 14

l Our event camera simulation is fully based on Monte Carlo path tracing, thus accurately reproducing physical behavior of light transport. l The key contribution of this study is thresholding for the residual of WLR, which avoids noise due to the lack of samples. l With the same sample budget, our method outperforms alternative baselines, such as ESIM and that combined with WLR. Conclusion Visit our GitHub: Code, Dataset, Results, etc. https://github.com/0V/ESIM-AD #1269, MA.PB: Denoising, 9th Oct., 11:00 - 12:30 15