3D Point Cloud Generation Using Adversarial Training for Large Scale Outdoor Scene

IGARSS2021
Title: 3D POINT CLOUD GENERATION USING ADVERSARIAL TRAINING FOR LARGE-SCALE OUTDOOR SCENE
Authors: Takayuki Shinohara, Haoyi Xiu, Masashi Matsuoka, Tokyo Institute of Technology, Japan

Abstract: Three-dimensional (3D) point clouds are becoming an important part of the geospatial domain. During research on 3D point clouds, deep-learning models have been widely used for the classification and segmentation of 3D point clouds observed by airborne LiDAR. However, most previous studies used discriminative models, whereas few studies used generative models. Specifically, one unsolved problem is the synthesis of large-scale 3D point clouds, such as those observed in outdoor scenes, because of the 3D point clouds’ complex geometric structure.
In this paper, we propose a generative model for generating large-scale 3D point clouds observed from airborne LiDAR. Generally, because the training process of the famous generative model called generative adversarial network (GAN) is unstable, we combine a variational autoencoder and GAN to generate a suitable 3D point cloud. We experimentally demonstrate that our framework can generate high-density 3D point clouds by using data from the 2018 IEEE GRSS Data Fusion Contest.

Transcript

1. 1
   3D Point Cloud Generation
   Using Adversarial Training
   for Large Scale Outdoor Scene
   Takayuki Shinohara, Haoyi Xiu, and Masashi Matsuoka
   Tokyo Institute of Technology
   15/July/2021
   Online, IGARSS2021
   

   View Slide

2. 2
   Tokyo Tech
   Outline
   1. Background and Objective
   2. Proposed Method
   3. Experimental Result
   4. Conclusion
   

   View Slide

3. 3
   Tokyo Tech
   1. Background and Objectives
   

   View Slide

4. 4
   Tokyo Tech
   Deep Learning (DL) on Point Cloud
   lDiscriminative Model
   n Classification
   n Semantic Segmentation
   n Object Detection
   lGenerative Model
   n Point Cloud Generation
   Many Researches
   We need to research generative models
   for airborne Point Cloud.
   Generative models for Point Cloud observed
   by airborne LiDAR (airborne Point Cloud)
   are not studied enough.
   

   View Slide

5. 5
   Tokyo Tech
   Generative model for Point Cloud
   lGenerative Adversarial Network
   Previous papers only generate simple objects
   Airborne Point Cloud has more
   complex objects than previous target
   latent 3d points (Achlioptas et al.)
   tree GAN (Shu et al.)
   Source: Shu et al.
   

   View Slide

6. 6
   Tokyo Tech
   Generative model for Point Cloud
   lGenerative Adversarial Network
   Previous papers only generate simple objects
   Airborne Point Cloud has more
   complex objects than previous target
   source: https://udayton.edu/engineering/research/centers/vision_lab/research/was_data_analysis_and_processing/dale.php
   Building
   Vegetation
   Ground
   Road
   

   View Slide

7. 7
   Tokyo Tech
   DL-based Generative Models
   l VAEs[Kingma&Welling, 2014]
   n pros😄
   l Clear objective
   function
   l Suitable training
   n cons😭
   l Blurred results
   l Simple distribution
   l GANs[Goodfellow et al., 2014]
   n pros😄
   l Clear results
   l Complex generation
   n cons😭
   l Not clear object
   function
   l Hard to train
   We propose VAE and GAN-based
   airborne Point Cloud generation
   with suitable training and clear results.
   Complex objects generation needs
   suitable training and clear results
   

   View Slide

8. 8
   Tokyo Tech
   2. Proposed Method
   

   View Slide

9. 9
   Tokyo Tech
   Overview of Our Method
   lVAE+GAN
   Reconstructed
   Fake Data
   G(𝑧)
   Input
   Data
   𝑥
   𝑧
   E G
   D
   Sampled from
   Real Data
   𝑥~𝑅𝑒𝑎𝑙
   Real/Fake
   PointNet++ TreeGAN
   PointNet
   VAE: Encoder and Generator
   GAN judges Real or Reconstructed Fake data
   VAE encodes input points
   into latent vector 𝑧 and
   reconstructs input data
   from latent vector
   GAN: Discriminator
   

   View Slide

10. 10
    Tokyo Tech
    Network: Encoder
    lPointNet++[Qi et al. 2017]-based
    𝑧 = 𝒩(𝜇(𝑥), 𝜎(𝑥))
    3
    N
    Input
    Data
    𝑥
    𝜇(𝑥)
    8,192
    4,096
    2,048
    1DCNN
    Downsampling( )
    Sampling
    Grouping
    Fully Connected
    𝜎(𝑥)
    𝑧 is sampled from
    estimated gaussian
    distribution
    Encoder estimates
    latent distribution
    Dow
    nsam
    pling
    and
    Convolution
    

    View Slide

11. 11
    Tokyo Tech
    Network: Generator
    lTreeGAN[Shu et al. 2017]-based
    𝑧 ∈ ℝ!"
    Upsampling
    and
    Convolution
    Fake
    Data
    latent vector
    from Encoder
    Reconstructed data
    𝐺 𝑧 = 𝑝# ∈ ℝ$×&
    Generator reconstructs input data
    from latent vector extracted by Encoder
    𝑝' ∈ ℝ'×& 𝑝( ∈ ℝ(×& 𝑝& ∈ ℝ)×& 𝑝#*' ∈ ℝ
    (
    $×&
    ⋯
    seed point Gradually Upsampling
    

    View Slide

12. 12
    Tokyo Tech
    Network: Discriminator
    lPointNet[Qi.et.al 2017]-based
    𝐺(𝑧)
    ́
    𝑥~𝑅𝑒𝑎𝑙
    Fake
    or
    Real
    Input
    …
    Discriminator judges Fake data or Real data
    and minimizes distribution between
    Fake data and Real data
    Fake
    Data
    Real
    Data MLP
    local features
    Max Pooling
    global
    features
    Randomly sampled
    from all of training data.
    Input
    MLP
    

    View Slide

13. 13
    Tokyo Tech
    Optimization
    lVAE
    n Reconstruction and Regularization
    𝐿!"# = 𝐿$%& + 𝐿$%'
    𝐿$%& =
    𝐿$%' = KL Divergence(𝑞(𝑧|𝑥) 𝑝 𝑧
    lGAN
    n Wasserstein Gradient Penurity loss
    Gen: 𝐿( = −𝔼) 𝐷 𝐺 𝑧
    Disc: 𝐿* = 𝔼) 𝐷 𝐺 𝑧 - 𝔼 ́
    ,~ℝ 𝐷 ́
    𝑥 + 𝜆'/𝔼0
    ,[( ∇0
    ,𝐷 B
    𝑥 1−1)1]
    Chamfer Distance Earth Mover Distance
    Penurity
    Close to Normal Distribution with 𝜇 = 0, 𝜎 = 1
    "
    !∈#!
    min
    $∈#"
    𝑥 − 𝑦 %
    % + "
    !∈#"
    min
    $∈#!
    𝑥 − 𝑦 %
    % + min
    #!→#"
    "
    ': !∈#!
    1
    2
    𝑥 − 𝜙(𝑥) %
    %
    

    View Slide

14. 14
    Tokyo Tech
    3. Experimental Results
    

    View Slide

15. 15
    Tokyo Tech
    Experimental Data
    lGRSS Data Fusion Contest 2018
    n Airborne LiDAR observation
    l Target Area
    l urban area
    l Building
    l Vegetation
    l Road
    l Training Patch
    l 25 m2
    l 2,048 points
    l 1,000 patches
    Training Patch
    side view
    top view
    25 m
    25 m
    Target Area
    

    View Slide

16. 16
    Tokyo Tech
    Generation Phase
    𝑧 Fake data
    𝐺(𝑧) ∈ ℝ#×%
    1. Random vector from estimated distribution
    𝑧 = 𝒩(𝜇, 𝜎), 𝜇 and 𝜎 are deRined in training process.
    2. Generating Fake data 𝐺(𝑧) via trained generator
    Fake data
    𝐺(𝑧) ∈ ℝ#×%
    Typical
    Fake data
    Typical
    Fake data
    Typical
    Fake data
    Typical
    Fake data
    3. Pick up generated Fake data including typical
    objects such as buildings and vegetation.
    Pick
    up
    

    View Slide

17. 17
    Tokyo Tech
    Generated Point Cloud
    Proposed GAN and VAE method generated
    better results than raw GAN model.
    

    View Slide

18. 18
    Tokyo Tech
    4. Conclusion and
    Future Work
    

    View Slide

19. 19
    Tokyo Tech
    Conclusion and Future Work
    lConclusion
    n We propose a generative model for Point
    Cloud observed by airborne LiDAR using
    VAE and GAN.
    n Our trained Generator was able to make
    fake point clouds clearly.
    lFuture work
    n Only Qualitative evaluation
    => Quantitative evaluation
    n Sparse fixed points
    => Dense point cloud generation
    n Traditional method
    => Change generator into recent architecture
    

    View Slide

20. 20
    Tokyo Tech
    Network: Generator
    lTree structure convolution
    latent vector
    Generated points
    Upsam
    pling
    with
    convolution
    Source: Shu et al.
    𝑧 ∈ ℝ!"
    

    View Slide

21. 22
    Tokyo Tech
    Overview of Our Method
    lVAE+GAN
    Reconstructed
    Fake Data
    G(𝑧)
    Input
    Data
    𝑥
    𝑧
    E G
    D
    Sampled from
    Real Data
    𝑥~𝑅𝑒𝑎𝑙
    Real/Fake
    PointNet++ TreeGAN
    PointNet
    VAE: Encoder and Generator
    GAN judges Real or Reconstructed Fake data
    latent
    vector
    VAE encodes input points
    into latent vector 𝑧 and
    reconstructs input data
    from latent vector
    GAN: Discriminator
    

    View Slide