1 3D Point Cloud Generation Using Adversarial Training for Large Scale Outdoor Scene Takayuki Shinohara, Haoyi Xiu, and Masashi Matsuoka Tokyo Institute of Technology 15/July/2021 Online, IGARSS2021

2 Tokyo Tech Outline 1. Background and Objective 2. Proposed Method 3. Experimental Result 4. Conclusion

3 Tokyo Tech 1. Background and Objectives

4 Tokyo Tech Deep Learning (DL) on Point Cloud lDiscriminative Model n Classification n Semantic Segmentation n Object Detection lGenerative Model n Point Cloud Generation Many Researches We need to research generative models for airborne Point Cloud. Generative models for Point Cloud observed by airborne LiDAR (airborne Point Cloud) are not studied enough.

5 Tokyo Tech Generative model for Point Cloud lGenerative Adversarial Network Previous papers only generate simple objects Airborne Point Cloud has more complex objects than previous target latent 3d points (Achlioptas et al.) tree GAN (Shu et al.) Source: Shu et al.

6 Tokyo Tech Generative model for Point Cloud lGenerative Adversarial Network Previous papers only generate simple objects Airborne Point Cloud has more complex objects than previous target source: https://udayton.edu/engineering/research/centers/vision_lab/research/was_data_analysis_and_processing/dale.php Building Vegetation Ground Road

7 Tokyo Tech DL-based Generative Models l VAEs[Kingma&Welling, 2014] n pros😄 l Clear objective function l Suitable training n cons😭 l Blurred results l Simple distribution l GANs[Goodfellow et al., 2014] n pros😄 l Clear results l Complex generation n cons😭 l Not clear object function l Hard to train We propose VAE and GAN-based airborne Point Cloud generation with suitable training and clear results. Complex objects generation needs suitable training and clear results

8 Tokyo Tech 2. Proposed Method

9 Tokyo Tech Overview of Our Method lVAE+GAN Reconstructed Fake Data G(𝑧) Input Data 𝑥 𝑧 E G D Sampled from Real Data 𝑥~𝑅𝑒𝑎𝑙 Real/Fake PointNet++ TreeGAN PointNet VAE: Encoder and Generator GAN judges Real or Reconstructed Fake data VAE encodes input points into latent vector 𝑧 and reconstructs input data from latent vector GAN: Discriminator

10 Tokyo Tech Network: Encoder lPointNet++[Qi et al. 2017]-based 𝑧 = 𝒩(𝜇(𝑥), 𝜎(𝑥)) 3 N Input Data 𝑥 𝜇(𝑥) 8,192 4,096 2,048 1DCNN Downsampling( ) Sampling Grouping Fully Connected 𝜎(𝑥) 𝑧 is sampled from estimated gaussian distribution Encoder estimates latent distribution Dow nsam pling and Convolution

11 Tokyo Tech Network: Generator lTreeGAN[Shu et al. 2017]-based 𝑧 ∈ ℝ!" Upsampling and Convolution Fake Data latent vector from Encoder Reconstructed data 𝐺 𝑧 = 𝑝# ∈ ℝ$×& Generator reconstructs input data from latent vector extracted by Encoder 𝑝' ∈ ℝ'×& 𝑝( ∈ ℝ(×& 𝑝& ∈ ℝ)×& 𝑝#*' ∈ ℝ ( $×& ⋯ seed point Gradually Upsampling

12 Tokyo Tech Network: Discriminator lPointNet[Qi.et.al 2017]-based 𝐺(𝑧) ́ 𝑥~𝑅𝑒𝑎𝑙 Fake or Real Input … Discriminator judges Fake data or Real data and minimizes distribution between Fake data and Real data Fake Data Real Data MLP local features Max Pooling global features Randomly sampled from all of training data. Input MLP

13 Tokyo Tech Optimization lVAE n Reconstruction and Regularization 𝐿!"# = 𝐿$%& + 𝐿$%' 𝐿$%& = 𝐿$%' = KL Divergence(𝑞(𝑧|𝑥) 𝑝 𝑧 lGAN n Wasserstein Gradient Penurity loss Gen: 𝐿( = −𝔼) 𝐷 𝐺 𝑧 Disc: 𝐿* = 𝔼) 𝐷 𝐺 𝑧 - 𝔼 ́ ,~ℝ 𝐷 ́ 𝑥 + 𝜆'/𝔼0 ,[( ∇0 ,𝐷 B 𝑥 1−1)1] Chamfer Distance Earth Mover Distance Penurity Close to Normal Distribution with 𝜇 = 0, 𝜎 = 1 " !∈#! min $∈#" 𝑥 − 𝑦 % % + " !∈#" min $∈#! 𝑥 − 𝑦 % % + min #!→#" " ': !∈#! 1 2 𝑥 − 𝜙(𝑥) % %

14 Tokyo Tech 3. Experimental Results

15 Tokyo Tech Experimental Data lGRSS Data Fusion Contest 2018 n Airborne LiDAR observation l Target Area l urban area l Building l Vegetation l Road l Training Patch l 25 m2 l 2,048 points l 1,000 patches Training Patch side view top view 25 m 25 m Target Area

16 Tokyo Tech Generation Phase 𝑧 Fake data 𝐺(𝑧) ∈ ℝ#×% 1. Random vector from estimated distribution 𝑧 = 𝒩(𝜇, 𝜎), 𝜇 and 𝜎 are deRined in training process. 2. Generating Fake data 𝐺(𝑧) via trained generator Fake data 𝐺(𝑧) ∈ ℝ#×% Typical Fake data Typical Fake data Typical Fake data Typical Fake data 3. Pick up generated Fake data including typical objects such as buildings and vegetation. Pick up

17 Tokyo Tech Generated Point Cloud Proposed GAN and VAE method generated better results than raw GAN model.

18 Tokyo Tech 4. Conclusion and Future Work

19 Tokyo Tech Conclusion and Future Work lConclusion n We propose a generative model for Point Cloud observed by airborne LiDAR using VAE and GAN. n Our trained Generator was able to make fake point clouds clearly. lFuture work n Only Qualitative evaluation => Quantitative evaluation n Sparse fixed points => Dense point cloud generation n Traditional method => Change generator into recent architecture

20 Tokyo Tech Network: Generator lTree structure convolution latent vector Generated points Upsam pling with convolution Source: Shu et al. 𝑧 ∈ ℝ!"

22 Tokyo Tech Overview of Our Method lVAE+GAN Reconstructed Fake Data G(𝑧) Input Data 𝑥 𝑧 E G D Sampled from Real Data 𝑥~𝑅𝑒𝑎𝑙 Real/Fake PointNet++ TreeGAN PointNet VAE: Encoder and Generator GAN judges Real or Reconstructed Fake data latent vector VAE encodes input points into latent vector 𝑧 and reconstructs input data from latent vector GAN: Discriminator