Do Deep Neural Networks Learn Full Waveform LiDAR Data?

I have talked about raw full waveform lidar data analysis using deep learning method called pointnet at Joint Student Seminar on Remote Sensing and Geoinformatics.

Transcript

1. Do Deep Neural Networks Learn
   Full Waveform LiDAR Data?
   Takayuki SHINOHARA
   Matsuoka Laboratory, Tokyo Institute of Technology
   2019/11/18 Joint Student Seminar on Remote Sensing and Geoinformatics
   

   View Slide

2. Background and Objective
   2
   

   View Slide

3. Full Waveform LiDAR
   nAdvantages
   • Recording the entire reflected
   signal “waveform” discretely.
   • Providing not only 3D point
   clouds, but also additional
   information
   about the target properties.
   nData Analysis
   • Costly and time consuming at
   manual processing.
   Automatic analysis method for "raw” waveform data is needed.
   3
   Cited from “Urban land cover classification using airborne LiDAR data: A review”
   

   View Slide

4. Raw Waveform analysis
   nDeep learning (S. Zorzi et al., 2019)
   • 2 stages supervised classification
   ‣ 1st stage: Waveform analysis by 1D CNN
   ⁃ Highly miss classification
   ‣ 2nd stage: Spatial analysis by 2D CNN
   ⁃ Raw classification results are converted
   to grid data
   ⁃ Prediction the class each grid.
   • Problems
   ‣ Spatial raw waveforms are not used
   in 1st stage.
   Þ Spatial learning method for
   raw waveform data are needed
   to improve performance. 4
   Waveform
   Spatial
   

   View Slide

5. Solve problem
   nPoint Net(Qi et.al, 2017)
   • One of the deep learning method for spatially irregular data.
   • Input order invariant network.
   nAuto Encoder as representation learning
   • Representation learning is data driven feature extraction method.
   • Auto Encoder is one of the method.
   • Auto Encoder can extract low dimensional latent vector from high
   dimensional data such as Image or some spatial data.
   Objective in this study:
   Using a deep learning method,
   a new representation learning method
   for spatially distributed raw full-waveform data.
   5
   

   View Slide

6. Proposed Method
   6
   

   View Slide

7. Proposed network (FWNetAE)
   nAuto Encoder for full waveform data
   • Encoder: PointNet based
   • Decoder: simple Multi Layer Perceptron(MLP)
   7
   62
   2,048
   …
   1D Conv
   features
   features
   features
   features
   features
   features
   …
   Input
   full waveform
   LiDAR
   Data
   Output
   full-waveform
   LiDAR
   Data
   Max Pool MLP 62
   2,048
   Latent vector
   PointNet based Encoder Decoder
   T-nets
   

   View Slide

8. Proposed network (FWNetAE)
   nEncoder: Point Net based Architecture
   • 1D CNN: Extract local features
   • MaxPool: Extract grobal features
   • T-nets: Extract rotation invariant features
   8
   62
   2,048
   …
   1D Conv
   features
   features
   features
   features
   features
   features
   …
   Input
   full waveform
   LiDAR
   Data
   Output
   full-waveform
   LiDAR
   Data
   Max Pool MLP 62
   2,048
   Latent vector
   PointNet based Encoder Decoder
   T-nets
   

   View Slide

9. Proposed network (FWNetAE)
   nDecoder: MLP based architecture
   • Fully connected layers to produce reconstructed data !
   of 2,048 × 62
   dimensions, i.e., the same as those of input data
   9
   62
   2,048
   …
   1D Conv
   features
   features
   features
   features
   features
   features
   …
   Input
   full waveform
   LiDAR
   Data
   Output
   full-waveform
   LiDAR
   Data
   Max Pool MLP 62
   2,048
   Latent vector
   PointNet based Encoder Decoder
   T-nets
   

   View Slide

10. Loss function
    nLoss function for reconstruction
    • FWNetAE aims at reconstructing target reconstructed data !
    .
    • Reconstruction loss functions.
    ‣ Spatial matching loss
    ‣ Waveform reconstruction
    10
    %&'()
    =
    1
    2N
    .
    /01
    2
    3 /
    − 6
    /
    7
    + /
    − 6
    /
    7), 1
    ?@A
    =
    1
    2N
    .
    /01
    2
    .
    B01
    C
    /,B
    − ̂
    /,B
    7
    . 2
    Optimization process
    Minimize these function
    

    View Slide

11. Experimental Results
    11
    

    View Slide

12. Dataset and Implementation
    nDataset
    • Dublin City Dataset
    ‣ Sample Size:
    ⁃ Train, Val, Test: 300,000, 100,000 100,000
    nImplementation and training results
    • Implementation
    ‣ Framework: PyTorch
    ‣ Hardware: 7th generation 3.8 GHz i7 CPU, 32 GB of RAM,
    and a Quadro P6000 GPU
    • Result
    ‣ Training time: approximately 9 hours
    ‣ Reconstruction error(test data): spatial=0.051, waveform=0.29
    12
    Used data
    

    View Slide

13. Reconstruction Results 1/2
    nSpatial reconstruction
    13
    A matching shape was observed.
    27
    Failure case
    

    View Slide

14. Reconstruction Results 2/2
    nWaveform reconstruction
    14
    28
    Failure case
    A matching shape was observed.
    

    View Slide

15. Latent Space Visualization
    nComparison of some method
    15
    Nonspatial AE
    PCA Proposed method
    (FWNetAE)
    Learnable algorithms
    Not Learnable
    

    View Slide

16. Conclusion and Future Study
    16
    

    View Slide

17. Conclusions and Future Study
    nConclusions
    • This paper presents a novel representation learning method for
    spatially distributed full-waveform data observed from an ALS using an
    AE-based architecture called FWNetAE.
    • The results demonstrate a generalization error for invisible test data.
    • Moreover, the FWNetAE encoded a meaningful latent vector and the
    decoders reconstructed the spatial geometry and its waveform value
    from the encoded latent vector.
    • However, the PointNet-based encoders could not extract features at
    various resolutions.
    nFuture Study
    • Modern Hieratical learning: PointNet++, Dynamic Graph CNN
    • Application for Supervised Learning
    17
    

    View Slide

18. Do Deep Neural Networks Learn
    Full Waveform LiDAR Data?
    Yes, they do.
    18
    

    View Slide