Upgrade to Pro — share decks privately, control downloads, hide ads and more …

3D Point Cloud Generation Using Adversarial Training for Large Scale Outdoor Scene

8c8a2735cb5acba72b16a56a85912c44?s=47 teddy
June 12, 2021

3D Point Cloud Generation Using Adversarial Training for Large Scale Outdoor Scene

IGARSS2021
Title: 3D POINT CLOUD GENERATION USING ADVERSARIAL TRAINING FOR LARGE-SCALE OUTDOOR SCENE
Authors: Takayuki Shinohara, Haoyi Xiu, Masashi Matsuoka, Tokyo Institute of Technology, Japan

Abstract: Three-dimensional (3D) point clouds are becoming an important part of the geospatial domain. During research on 3D point clouds, deep-learning models have been widely used for the classification and segmentation of 3D point clouds observed by airborne LiDAR. However, most previous studies used discriminative models, whereas few studies used generative models. Specifically, one unsolved problem is the synthesis of large-scale 3D point clouds, such as those observed in outdoor scenes, because of the 3D point clouds’ complex geometric structure.
In this paper, we propose a generative model for generating large-scale 3D point clouds observed from airborne LiDAR. Generally, because the training process of the famous generative model called generative adversarial network (GAN) is unstable, we combine a variational autoencoder and GAN to generate a suitable 3D point cloud. We experimentally demonstrate that our framework can generate high-density 3D point clouds by using data from the 2018 IEEE GRSS Data Fusion Contest.

8c8a2735cb5acba72b16a56a85912c44?s=128

teddy

June 12, 2021
Tweet

Transcript

  1. 1 3D Point Cloud Generation Using Adversarial Training for Large

    Scale Outdoor Scene Takayuki Shinohara, Haoyi Xiu, and Masashi Matsuoka Tokyo Institute of Technology 15/July/2021 Online, IGARSS2021
  2. 2 Tokyo Tech Outline 1. Background and Objective 2. Proposed

    Method 3. Experimental Result 4. Conclusion
  3. 3 Tokyo Tech 1. Background and Objectives

  4. 4 Tokyo Tech Deep Learning (DL) on Point Cloud lDiscriminative

    Model n Classification n Semantic Segmentation n Object Detection lGenerative Model n Point Cloud Generation Many Researches We need to research generative models for airborne Point Cloud. Generative models for Point Cloud observed by airborne LiDAR (airborne Point Cloud) are not studied enough.
  5. 5 Tokyo Tech Generative model for Point Cloud lGenerative Adversarial

    Network Previous papers only generate simple objects Airborne Point Cloud has more complex objects than previous target latent 3d points (Achlioptas et al.) tree GAN (Shu et al.) Source: Shu et al.
  6. 6 Tokyo Tech Generative model for Point Cloud lGenerative Adversarial

    Network Previous papers only generate simple objects Airborne Point Cloud has more complex objects than previous target source: https://udayton.edu/engineering/research/centers/vision_lab/research/was_data_analysis_and_processing/dale.php Building Vegetation Ground Road
  7. 7 Tokyo Tech DL-based Generative Models l VAEs[Kingma&Welling, 2014] n

    pros😄 l Clear objective function l Suitable training n cons😭 l Blurred results l Simple distribution l GANs[Goodfellow et al., 2014] n pros😄 l Clear results l Complex generation n cons😭 l Not clear object function l Hard to train We propose VAE and GAN-based airborne Point Cloud generation with suitable training and clear results. Complex objects generation needs suitable training and clear results
  8. 8 Tokyo Tech 2. Proposed Method

  9. 9 Tokyo Tech Overview of Our Method lVAE+GAN Reconstructed Fake

    Data G(𝑧) Input Data 𝑥 𝑧 E G D Sampled from Real Data 𝑥~𝑅𝑒𝑎𝑙 Real/Fake PointNet++ TreeGAN PointNet VAE: Encoder and Generator GAN judges Real or Reconstructed Fake data VAE encodes input points into latent vector 𝑧 and reconstructs input data from latent vector GAN: Discriminator
  10. 10 Tokyo Tech Network: Encoder lPointNet++[Qi et al. 2017]-based 𝑧

    = 𝒩(𝜇(𝑥), 𝜎(𝑥)) 3 N Input Data 𝑥 𝜇(𝑥) 8,192 4,096 2,048 1DCNN Downsampling( ) Sampling Grouping Fully Connected 𝜎(𝑥) 𝑧 is sampled from estimated gaussian distribution Encoder estimates latent distribution Dow nsam pling and Convolution
  11. 11 Tokyo Tech Network: Generator lTreeGAN[Shu et al. 2017]-based 𝑧

    ∈ ℝ!" Upsampling and Convolution Fake Data latent vector from Encoder Reconstructed data 𝐺 𝑧 = 𝑝# ∈ ℝ$×& Generator reconstructs input data from latent vector extracted by Encoder 𝑝' ∈ ℝ'×& 𝑝( ∈ ℝ(×& 𝑝& ∈ ℝ)×& 𝑝#*' ∈ ℝ ( $×& ⋯ seed point Gradually Upsampling
  12. 12 Tokyo Tech Network: Discriminator lPointNet[Qi.et.al 2017]-based 𝐺(𝑧) ́ 𝑥~𝑅𝑒𝑎𝑙

    Fake or Real Input … Discriminator judges Fake data or Real data and minimizes distribution between Fake data and Real data Fake Data Real Data MLP local features Max Pooling global features Randomly sampled from all of training data. Input MLP
  13. 13 Tokyo Tech Optimization lVAE n Reconstruction and Regularization 𝐿!"#

    = 𝐿$%& + 𝐿$%' 𝐿$%& = 𝐿$%' = KL Divergence(𝑞(𝑧|𝑥) 𝑝 𝑧 lGAN n Wasserstein Gradient Penurity loss Gen: 𝐿( = −𝔼) 𝐷 𝐺 𝑧 Disc: 𝐿* = 𝔼) 𝐷 𝐺 𝑧 - 𝔼 ́ ,~ℝ 𝐷 ́ 𝑥 + 𝜆'/𝔼0 ,[( ∇0 ,𝐷 B 𝑥 1−1)1] Chamfer Distance Earth Mover Distance Penurity Close to Normal Distribution with 𝜇 = 0, 𝜎 = 1 " !∈#! min $∈#" 𝑥 − 𝑦 % % + " !∈#" min $∈#! 𝑥 − 𝑦 % % + min #!→#" " ': !∈#! 1 2 𝑥 − 𝜙(𝑥) % %
  14. 14 Tokyo Tech 3. Experimental Results

  15. 15 Tokyo Tech Experimental Data lGRSS Data Fusion Contest 2018

    n Airborne LiDAR observation l Target Area l urban area l Building l Vegetation l Road l Training Patch l 25 m2 l 2,048 points l 1,000 patches Training Patch side view top view 25 m 25 m Target Area
  16. 16 Tokyo Tech Generation Phase 𝑧 Fake data 𝐺(𝑧) ∈

    ℝ#×% 1. Random vector from estimated distribution 𝑧 = 𝒩(𝜇, 𝜎), 𝜇 and 𝜎 are deRined in training process. 2. Generating Fake data 𝐺(𝑧) via trained generator Fake data 𝐺(𝑧) ∈ ℝ#×% Typical Fake data Typical Fake data Typical Fake data Typical Fake data 3. Pick up generated Fake data including typical objects such as buildings and vegetation. Pick up
  17. 17 Tokyo Tech Generated Point Cloud Proposed GAN and VAE

    method generated better results than raw GAN model.
  18. 18 Tokyo Tech 4. Conclusion and Future Work

  19. 19 Tokyo Tech Conclusion and Future Work lConclusion n We

    propose a generative model for Point Cloud observed by airborne LiDAR using VAE and GAN. n Our trained Generator was able to make fake point clouds clearly. lFuture work n Only Qualitative evaluation => Quantitative evaluation n Sparse fixed points => Dense point cloud generation n Traditional method => Change generator into recent architecture
  20. 20 Tokyo Tech Network: Generator lTree structure convolution latent vector

    Generated points Upsam pling with convolution Source: Shu et al. 𝑧 ∈ ℝ!"
  21. 22 Tokyo Tech Overview of Our Method lVAE+GAN Reconstructed Fake

    Data G(𝑧) Input Data 𝑥 𝑧 E G D Sampled from Real Data 𝑥~𝑅𝑒𝑎𝑙 Real/Fake PointNet++ TreeGAN PointNet VAE: Encoder and Generator GAN judges Real or Reconstructed Fake data latent vector VAE encodes input points into latent vector 𝑧 and reconstructs input data from latent vector GAN: Discriminator