ACM SIGSPATIAL 2020
for Full Waveform LiDAR data
using Local and Hierarchical Global
Takayuki Shinohara, Haoyi Xiu and Masashi Matsuoka
Tokyo Institute of Technology
ACM SIGSPATIAL 2020 2020/11/06
Full waveform(FW) LiDAR
• Recording the entire
reflected signal as
• Definition of FW data
⁃ x, y, z, and waveform
• 3D point clouds and
regarding the target
The shape of waveform and power of the backscatter
are related to characteristics of the targets.
Cited from “Urban land cover classification using airborne LiDAR data: A review”
Previous Methods: Converting FW data
nHand crafted feature[Lai et al. (2019)]
• Point Clouds and features from waveform
• Waveform information improves the semantic
(Land Use/Land Cover classification)
n2D image[Zorzi et al. (2019)]
• Waveform and height information converted to 2D.
• Deep Learning(CNN) based methods can easily
deal with 2D images.
• Spatial feature extraction improves semantic
Direct and spatial feature extraction method
without converting to feature or grid is needed
Sem. Seg. for FW LiDAR Data
nWaveform or Geometric information
• Local Group
⁃ Waveform improves the segmentation
‣ e.g. Vegetations have many return peaks.
• Global Group
⁃ An Large area of spatial/geometric context
improves the segmentation performance.
‣ e.g. Roads have a uniform distribution
of similar waveforms.
OBJECTIVE: We propose a deep learning-based
method combining local and global feature
extractions for FW data sem. seg.
Images are cited from
Awange J., Kiema J. (2019) Light Detection And Ranging (LiDAR). In: Environmental Geoinformatics. Environmental Science and Engineering. Springer, Cham. https://doi.org/10.1007/978-3-030-03017-9_21
2. Proposed Method
nSemantic segmentation task for FW data
• Input: point (x, y, z) and waveform
⁃ ∈ ℝ!×#, $ = $, $, $, $ , $ ∈ ℝ#
‣ K: #points, M: x, y, z and length of waveform
• Output: segmentation result
⁃ ∈ ℝ!×&, $ = $,', $,(, ⋯ , $,& , $ ∈ ℝ&
‣ K: #points, C: #classes
Our model predict segmentation results
from input points and waveforms.
nLocal Module: Simple 1D CNN
Class prediction from local waveform information.
Simple weight shared 1D CNN
nGlobal Module: PointNet++ based model
Hierarchical subsample and upsample strategy to
extract global geometric context information.
• Chose the typical
points by subsample.
• Convolution needs
Providing waveform relationships in large area.
• Recovering the number
of points from subsampled
• Skip connection provides
vanished high res. info..
nConvolution for waveform
• Weight shared 1D CNN
for grouped waveform
• Waveform is defined as
sequential data like
• 1D CNN are widely used
in audio recognition.
• MaxPool operation
are used the same as
Calculating waveform features for grouped data.
'$"(#$ + %&'
= − $
log + ∑)*+
• Optimizer: ADAM with learning rate 0.005
⁃ Radius ball=3, 5, and 15
⁃ #neighbors = 16, 32, and 64
⁃ #features = 256, 512, and 1024
• Hardware: P100 on TSUBAME3.0
Loss for Local module and global module
at same time
each loss is weighted cross entropy
class ratio aware weight
Prediction from local and global module
nCombine outputs from local and global
The class which waveform improves use local module
The class which geometry improves use global module
3. Experimental Results
• FW data published by Zorzi et.al,2019
⁃ Test: red area
⁃ Ground, Build., Veg., Power Line, Trans. Tower, and Street Path
nDivided into 2 group
• Global group
⁃ Geometry is effective.
⁃ Ground, Build., Street Path
• Local group
⁃ Local features are effective.
⁃ Veg., PowerLine, Trans.Tower
[Zorzi et al., 2019]
As an overall trend, our model predicted correct classes.
Border area and street path are misclassified.
Our method achieved the highest f1 scores.
• Previous studies
⁃ Our model achieved higher scores.
• Comparison with other deep learning methods
⁃ Our local and global strategy achieved higher scores.
nComparisons of PointNet++ based baselines
⁃ PointNet++ w/ waveform vs. PointNet++ w/o waveform
• Hierarchical feature extraction
⁃ PointNet++ w/ Hierarchical vs. PointNet++ w/o Hierarchical
• Local module
⁃ PointNet++ w/ local vs. PointNet++ w/o local
We showed the effetiveness of PointNet++ based
model, waveform information, and local module
The Effect of Local Module
nAcceptance rate from local module
• Prediction step
⁃ Rule-based predictions using
the output of local module and global module.
⁃ Local Module predicts local group (Vegetation,
Power Line, Transmission Tower), which we
made the assumption that the local waveform
information is effective.
Our local module was able to predict local group
Conclusion and Future Study
• We have proposed a deep learning-based semantic
segmentation model for full-waveform LiDAR data.
• Our model consists of the local module which
predicts class from each waveform, and the global
module which predicts class from geometry.
• Experimental results showed that our model
predicts higher accuracy than previous methods.
nUnsolved Problems and Future Works
• Explicit Geometric information
⁃ Combine geometric and waveform feature extractions
• Heuristic combination method
⁃ Changing heuristic rule to learnable functions