Semantic Segmentation for Full Waveform LiDAR data using Local and Hierarchical Global Feature Extraction Takayuki Shinohara, Haoyi Xiu and Masashi Matsuoka Tokyo Institute of Technology 1 ACM SIGSPATIAL 2020 2020/11/06 online

1. Introduction 2

Full waveform(FW) LiDAR nAdvantages • Recording the entire reflected signal as “waveform” discretely. • Definition of FW data ⁃ x, y, z, and waveform • 3D point clouds and additional information regarding the target properties. The shape of waveform and power of the backscatter are related to characteristics of the targets. 3 Cited from “Urban land cover classification using airborne LiDAR data: A review”

Previous Methods: Converting FW data nHand crafted feature[Lai et al. (2019)] • Point Clouds and features from waveform • Waveform information improves the semantic segmentation task (Land Use/Land Cover classification) n2D image[Zorzi et al. (2019)] • Waveform and height information converted to 2D. • Deep Learning(CNN) based methods can easily deal with 2D images. • Spatial feature extraction improves semantic segmentation. 4 Direct and spatial feature extraction method without converting to feature or grid is needed

Sem. Seg. for FW LiDAR Data nWaveform or Geometric information • Local Group ⁃ Waveform improves the segmentation performance. ‣ e.g. Vegetations have many return peaks. • Global Group ⁃ An Large area of spatial/geometric context improves the segmentation performance. ‣ e.g. Roads have a uniform distribution of similar waveforms. OBJECTIVE: We propose a deep learning-based method combining local and global feature extractions for FW data sem. seg. 5 Images are cited from Awange J., Kiema J. (2019) Light Detection And Ranging (LiDAR). In: Environmental Geoinformatics. Environmental Science and Engineering. Springer, Cham. https://doi.org/10.1007/978-3-030-03017-9_21

2. Proposed Method 6

Problem Definition nSemantic segmentation task for FW data • Input: point (x, y, z) and waveform ⁃ ∈ ℝ!×#, $ = $, $, $, $ , $ ∈ ℝ# ‣ K: #points, M: x, y, z and length of waveform • Output: segmentation result ⁃ ∈ ℝ!×&, $ = $,', $,(, ⋯ , $,& , $ ∈ ℝ& ‣ K: #points, C: #classes 7 Our model predict segmentation results from input points and waveforms. Input Output

Network Architecture nLocal Module: Simple 1D CNN 8 Class prediction from local waveform information. Simple weight shared 1D CNN

Network Architecture nGlobal Module: PointNet++ based model 9 Hierarchical subsample and upsample strategy to extract global geometric context information. Subsample step Upsample step

Global Module nSubsample • Chose the typical points by subsample. nDefine neighbors • Convolution needs neighbor points. 10 Providing waveform relationships in large area. nUpsample • Recovering the number of points from subsampled points. • Skip connection provides vanished high res. info..

Convolutional Operation nConvolution for waveform • Weight shared 1D CNN for grouped waveform or features. • Waveform is defined as sequential data like audio signals. • 1D CNN are widely used in audio recognition. • MaxPool operation after convolution are used the same as PointNet++. 11 Calculating waveform features for grouped data.

Optimization nLoss function !"!#$ = %&' '$"(#$ + %&' $")#$ !"# = − $ $ % $ & ' & ∗ $,& log $,& ) = 1/ ) log + ∑)*+ , ) nTraining detail • Optimizer: ADAM with learning rate 0.005 • Convolution ⁃ Radius ball=3, 5, and 15 ⁃ #neighbors = 16, 32, and 64 ⁃ #features = 256, 512, and 1024 • Hardware: P100 on TSUBAME3.0 12 Loss for Local module and global module at same time each loss is weighted cross entropy class ratio aware weight

Prediction from local and global module nCombine outputs from local and global 13 The class which waveform improves use local module The class which geometry improves use global module

3. Experimental Results 14

Dataset nUsed data • FW data published by Zorzi et.al,2019 ⁃ Test: red area • Class ⁃ Ground, Build., Veg., Power Line, Trans. Tower, and Street Path nDivided into 2 group • Global group ⁃ Geometry is effective. ⁃ Ground, Build., Street Path • Local group ⁃ Local features are effective. ⁃ Veg., PowerLine, Trans.Tower 15 600 m 70 m 24 m test [Zorzi et al., 2019]

Qualitative Evaluation 16 As an overall trend, our model predicted correct classes. Border area and street path are misclassified. Boundary area Street Path Boundary area

Quantitative Evaluation 17 Our method achieved the highest f1 scores. nUsed data • Previous studies ⁃ Our model achieved higher scores. • Comparison with other deep learning methods ⁃ Our local and global strategy achieved higher scores.

Ablation Study nComparisons of PointNet++ based baselines • Waveform ⁃ PointNet++ w/ waveform vs. PointNet++ w/o waveform • Hierarchical feature extraction ⁃ PointNet++ w/ Hierarchical vs. PointNet++ w/o Hierarchical • Local module ⁃ PointNet++ w/ local vs. PointNet++ w/o local 18 We showed the effetiveness of PointNet++ based model, waveform information, and local module

The Effect of Local Module nAcceptance rate from local module • Prediction step ⁃ Rule-based predictions using the output of local module and global module. ⁃ Local Module predicts local group (Vegetation, Power Line, Transmission Tower), which we made the assumption that the local waveform information is effective. 19 Our local module was able to predict local group

4. Conclusion 20

Conclusion and Future Study nConclusion • We have proposed a deep learning-based semantic segmentation model for full-waveform LiDAR data. • Our model consists of the local module which predicts class from each waveform, and the global module which predicts class from geometry. • Experimental results showed that our model predicts higher accuracy than previous methods. nUnsolved Problems and Future Works • Explicit Geometric information ⁃ Combine geometric and waveform feature extractions • Heuristic combination method ⁃ Changing heuristic rule to learnable functions 21