Slide 1

Slide 1 text

Semantic Segmentation for Full Waveform LiDAR data using Local and Hierarchical Global Feature Extraction Takayuki Shinohara, Haoyi Xiu and Masashi Matsuoka Tokyo Institute of Technology 1 ACM SIGSPATIAL 2020 2020/11/06 online

Slide 2

Slide 2 text

1. Introduction 2

Slide 3

Slide 3 text

Full waveform(FW) LiDAR nAdvantages • Recording the entire reflected signal as “waveform” discretely. • Definition of FW data ⁃ x, y, z, and waveform • 3D point clouds and additional information regarding the target properties. The shape of waveform and power of the backscatter are related to characteristics of the targets. 3 Cited from “Urban land cover classification using airborne LiDAR data: A review”

Slide 4

Slide 4 text

Previous Methods: Converting FW data nHand crafted feature[Lai et al. (2019)] • Point Clouds and features from waveform • Waveform information improves the semantic segmentation task (Land Use/Land Cover classification) n2D image[Zorzi et al. (2019)] • Waveform and height information converted to 2D. • Deep Learning(CNN) based methods can easily deal with 2D images. • Spatial feature extraction improves semantic segmentation. 4 Direct and spatial feature extraction method without converting to feature or grid is needed

Slide 5

Slide 5 text

Sem. Seg. for FW LiDAR Data nWaveform or Geometric information • Local Group ⁃ Waveform improves the segmentation performance. ‣ e.g. Vegetations have many return peaks. • Global Group ⁃ An Large area of spatial/geometric context improves the segmentation performance. ‣ e.g. Roads have a uniform distribution of similar waveforms. OBJECTIVE: We propose a deep learning-based method combining local and global feature extractions for FW data sem. seg. 5 Images are cited from Awange J., Kiema J. (2019) Light Detection And Ranging (LiDAR). In: Environmental Geoinformatics. Environmental Science and Engineering. Springer, Cham. https://doi.org/10.1007/978-3-030-03017-9_21

Slide 6

Slide 6 text

2. Proposed Method 6

Slide 7

Slide 7 text

Problem Definition nSemantic segmentation task for FW data • Input: point (x, y, z) and waveform ⁃ ∈ ℝ!×#, $ = $, $, $, $ , $ ∈ ℝ# ‣ K: #points, M: x, y, z and length of waveform • Output: segmentation result ⁃ ∈ ℝ!×&, $ = $,', $,(, ⋯ , $,& , $ ∈ ℝ& ‣ K: #points, C: #classes 7 Our model predict segmentation results from input points and waveforms. Input Output

Slide 8

Slide 8 text

Network Architecture nLocal Module: Simple 1D CNN 8 Class prediction from local waveform information. Simple weight shared 1D CNN

Slide 9

Slide 9 text

Network Architecture nGlobal Module: PointNet++ based model 9 Hierarchical subsample and upsample strategy to extract global geometric context information. Subsample step Upsample step

Slide 10

Slide 10 text

Global Module nSubsample • Chose the typical points by subsample. nDefine neighbors • Convolution needs neighbor points. 10 Providing waveform relationships in large area. nUpsample • Recovering the number of points from subsampled points. • Skip connection provides vanished high res. info..

Slide 11

Slide 11 text

Convolutional Operation nConvolution for waveform • Weight shared 1D CNN for grouped waveform or features. • Waveform is defined as sequential data like audio signals. • 1D CNN are widely used in audio recognition. • MaxPool operation after convolution are used the same as PointNet++. 11 Calculating waveform features for grouped data.

Slide 12

Slide 12 text

Optimization nLoss function !"!#$ = %&' '$"(#$ + %&' $")#$ !"# = − $ $ % $ & ' & ∗ $,& log $,& ) = 1/ ) log + ∑)*+ , ) nTraining detail • Optimizer: ADAM with learning rate 0.005 • Convolution ⁃ Radius ball=3, 5, and 15 ⁃ #neighbors = 16, 32, and 64 ⁃ #features = 256, 512, and 1024 • Hardware: P100 on TSUBAME3.0 12 Loss for Local module and global module at same time each loss is weighted cross entropy class ratio aware weight

Slide 13

Slide 13 text

Prediction from local and global module nCombine outputs from local and global 13 The class which waveform improves use local module The class which geometry improves use global module

Slide 14

Slide 14 text

3. Experimental Results 14

Slide 15

Slide 15 text

Dataset nUsed data • FW data published by Zorzi et.al,2019 ⁃ Test: red area • Class ⁃ Ground, Build., Veg., Power Line, Trans. Tower, and Street Path nDivided into 2 group • Global group ⁃ Geometry is effective. ⁃ Ground, Build., Street Path • Local group ⁃ Local features are effective. ⁃ Veg., PowerLine, Trans.Tower 15 600 m 70 m 24 m test [Zorzi et al., 2019]

Slide 16

Slide 16 text

Qualitative Evaluation 16 As an overall trend, our model predicted correct classes. Border area and street path are misclassified. Boundary area Street Path Boundary area

Slide 17

Slide 17 text

Quantitative Evaluation 17 Our method achieved the highest f1 scores. nUsed data • Previous studies ⁃ Our model achieved higher scores. • Comparison with other deep learning methods ⁃ Our local and global strategy achieved higher scores.

Slide 18

Slide 18 text

Ablation Study nComparisons of PointNet++ based baselines • Waveform ⁃ PointNet++ w/ waveform vs. PointNet++ w/o waveform • Hierarchical feature extraction ⁃ PointNet++ w/ Hierarchical vs. PointNet++ w/o Hierarchical • Local module ⁃ PointNet++ w/ local vs. PointNet++ w/o local 18 We showed the effetiveness of PointNet++ based model, waveform information, and local module

Slide 19

Slide 19 text

The Effect of Local Module nAcceptance rate from local module • Prediction step ⁃ Rule-based predictions using the output of local module and global module. ⁃ Local Module predicts local group (Vegetation, Power Line, Transmission Tower), which we made the assumption that the local waveform information is effective. 19 Our local module was able to predict local group

Slide 20

Slide 20 text

4. Conclusion 20

Slide 21

Slide 21 text

Conclusion and Future Study nConclusion • We have proposed a deep learning-based semantic segmentation model for full-waveform LiDAR data. • Our model consists of the local module which predicts class from each waveform, and the global module which predicts class from geometry. • Experimental results showed that our model predicts higher accuracy than previous methods. nUnsolved Problems and Future Works • Explicit Geometric information ⁃ Combine geometric and waveform feature extractions • Heuristic combination method ⁃ Changing heuristic rule to learnable functions 21