Slide 1

Slide 1 text

1 3D Point Cloud Generation Using Adversarial Training for Large Scale Outdoor Scene Takayuki Shinohara, Haoyi Xiu, and Masashi Matsuoka Tokyo Institute of Technology 15/July/2021 Online, IGARSS2021

Slide 2

Slide 2 text

2 Tokyo Tech Outline 1. Background and Objective 2. Proposed Method 3. Experimental Result 4. Conclusion

Slide 3

Slide 3 text

3 Tokyo Tech 1. Background and Objectives

Slide 4

Slide 4 text

4 Tokyo Tech Deep Learning (DL) on Point Cloud lDiscriminative Model n Classification n Semantic Segmentation n Object Detection lGenerative Model n Point Cloud Generation Many Researches We need to research generative models for airborne Point Cloud. Generative models for Point Cloud observed by airborne LiDAR (airborne Point Cloud) are not studied enough.

Slide 5

Slide 5 text

5 Tokyo Tech Generative model for Point Cloud lGenerative Adversarial Network Previous papers only generate simple objects Airborne Point Cloud has more complex objects than previous target latent 3d points (Achlioptas et al.) tree GAN (Shu et al.) Source: Shu et al.

Slide 6

Slide 6 text

6 Tokyo Tech Generative model for Point Cloud lGenerative Adversarial Network Previous papers only generate simple objects Airborne Point Cloud has more complex objects than previous target source: https://udayton.edu/engineering/research/centers/vision_lab/research/was_data_analysis_and_processing/dale.php Building Vegetation Ground Road

Slide 7

Slide 7 text

7 Tokyo Tech DL-based Generative Models l VAEs[Kingma&Welling, 2014] n prosπŸ˜„ l Clear objective function l Suitable training n cons😭 l Blurred results l Simple distribution l GANs[Goodfellow et al., 2014] n prosπŸ˜„ l Clear results l Complex generation n cons😭 l Not clear object function l Hard to train We propose VAE and GAN-based airborne Point Cloud generation with suitable training and clear results. Complex objects generation needs suitable training and clear results

Slide 8

Slide 8 text

8 Tokyo Tech 2. Proposed Method

Slide 9

Slide 9 text

9 Tokyo Tech Overview of Our Method lVAE+GAN Reconstructed Fake Data G(𝑧) Input Data π‘₯ 𝑧 E G D Sampled from Real Data π‘₯~π‘…π‘’π‘Žπ‘™ Real/Fake PointNet++ TreeGAN PointNet VAE: Encoder and Generator GAN judges Real or Reconstructed Fake data VAE encodes input points into latent vector 𝑧 and reconstructs input data from latent vector GAN: Discriminator

Slide 10

Slide 10 text

10 Tokyo Tech Network: Encoder lPointNet++[Qi et al. 2017]-based 𝑧 = 𝒩(πœ‡(π‘₯), 𝜎(π‘₯)) 3 N Input Data π‘₯ πœ‡(π‘₯) 8,192 4,096 2,048 1DCNN Downsampling( ) Sampling Grouping Fully Connected 𝜎(π‘₯) 𝑧 is sampled from estimated gaussian distribution Encoder estimates latent distribution Dow nsam pling and Convolution

Slide 11

Slide 11 text

11 Tokyo Tech Network: Generator lTreeGAN[Shu et al. 2017]-based 𝑧 ∈ ℝ!" Upsampling and Convolution Fake Data latent vector from Encoder Reconstructed data 𝐺 𝑧 = 𝑝# ∈ ℝ$Γ—& Generator reconstructs input data from latent vector extracted by Encoder 𝑝' ∈ ℝ'Γ—& 𝑝( ∈ ℝ(Γ—& 𝑝& ∈ ℝ)Γ—& 𝑝#*' ∈ ℝ ( $Γ—& β‹― seed point Gradually Upsampling

Slide 12

Slide 12 text

12 Tokyo Tech Network: Discriminator lPointNet[Qi.et.al 2017]-based 𝐺(𝑧) ́ π‘₯~π‘…π‘’π‘Žπ‘™ Fake or Real Input … Discriminator judges Fake data or Real data and minimizes distribution between Fake data and Real data Fake Data Real Data MLP local features Max Pooling global features Randomly sampled from all of training data. Input MLP

Slide 13

Slide 13 text

13 Tokyo Tech Optimization lVAE n Reconstruction and Regularization 𝐿!"# = 𝐿$%& + 𝐿$%' 𝐿$%& = 𝐿$%' = KL Divergence(π‘ž(𝑧|π‘₯) 𝑝 𝑧 lGAN n Wasserstein Gradient Penurity loss Gen: 𝐿( = βˆ’π”Ό) 𝐷 𝐺 𝑧 Disc: 𝐿* = 𝔼) 𝐷 𝐺 𝑧 - 𝔼 ́ ,~ℝ 𝐷 ́ π‘₯ + πœ†'/𝔼0 ,[( βˆ‡0 ,𝐷 B π‘₯ 1βˆ’1)1] Chamfer Distance Earth Mover Distance Penurity Close to Normal Distribution with πœ‡ = 0, 𝜎 = 1 " !∈#! min $∈#" π‘₯ βˆ’ 𝑦 % % + " !∈#" min $∈#! π‘₯ βˆ’ 𝑦 % % + min #!β†’#" " ': !∈#! 1 2 π‘₯ βˆ’ πœ™(π‘₯) % %

Slide 14

Slide 14 text

14 Tokyo Tech 3. Experimental Results

Slide 15

Slide 15 text

15 Tokyo Tech Experimental Data lGRSS Data Fusion Contest 2018 n Airborne LiDAR observation l Target Area l urban area l Building l Vegetation l Road l Training Patch l 25 m2 l 2,048 points l 1,000 patches Training Patch side view top view 25 m 25 m Target Area

Slide 16

Slide 16 text

16 Tokyo Tech Generation Phase 𝑧 Fake data 𝐺(𝑧) ∈ ℝ#Γ—% 1. Random vector from estimated distribution 𝑧 = 𝒩(πœ‡, 𝜎), πœ‡ and 𝜎 are deRined in training process. 2. Generating Fake data 𝐺(𝑧) via trained generator Fake data 𝐺(𝑧) ∈ ℝ#Γ—% Typical Fake data Typical Fake data Typical Fake data Typical Fake data 3. Pick up generated Fake data including typical objects such as buildings and vegetation. Pick up

Slide 17

Slide 17 text

17 Tokyo Tech Generated Point Cloud Proposed GAN and VAE method generated better results than raw GAN model.

Slide 18

Slide 18 text

18 Tokyo Tech 4. Conclusion and Future Work

Slide 19

Slide 19 text

19 Tokyo Tech Conclusion and Future Work lConclusion n We propose a generative model for Point Cloud observed by airborne LiDAR using VAE and GAN. n Our trained Generator was able to make fake point clouds clearly. lFuture work n Only Qualitative evaluation => Quantitative evaluation n Sparse fixed points => Dense point cloud generation n Traditional method => Change generator into recent architecture

Slide 20

Slide 20 text

20 Tokyo Tech Network: Generator lTree structure convolution latent vector Generated points Upsam pling with convolution Source: Shu et al. 𝑧 ∈ ℝ!"

Slide 21

Slide 21 text

22 Tokyo Tech Overview of Our Method lVAE+GAN Reconstructed Fake Data G(𝑧) Input Data π‘₯ 𝑧 E G D Sampled from Real Data π‘₯~π‘…π‘’π‘Žπ‘™ Real/Fake PointNet++ TreeGAN PointNet VAE: Encoder and Generator GAN judges Real or Reconstructed Fake data latent vector VAE encodes input points into latent vector 𝑧 and reconstructs input data from latent vector GAN: Discriminator