Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Semantic Segmentation for Full Waveform LiDAR data using Local and Hierarchical Global Feature Extraction

teddy
November 06, 2020

Semantic Segmentation for Full Waveform LiDAR data using Local and Hierarchical Global Feature Extraction

teddy

November 06, 2020
Tweet

More Decks by teddy

Other Decks in Research

Transcript

  1. Semantic Segmentation
    for Full Waveform LiDAR data
    using Local and Hierarchical Global
    Feature Extraction
    Takayuki Shinohara, Haoyi Xiu and Masashi Matsuoka
    Tokyo Institute of Technology
    1
    ACM SIGSPATIAL 2020 2020/11/06
    online

    View full-size slide

  2. 1. Introduction
    2

    View full-size slide

  3. Full waveform(FW) LiDAR
    nAdvantages
    • Recording the entire
    reflected signal as
    “waveform” discretely.
    • Definition of FW data
    ⁃ x, y, z, and waveform
    • 3D point clouds and
    additional information
    regarding the target
    properties.
    The shape of waveform and power of the backscatter
    are related to characteristics of the targets.
    3
    Cited from “Urban land cover classification using airborne LiDAR data: A review”

    View full-size slide

  4. Previous Methods: Converting FW data
    nHand crafted feature[Lai et al. (2019)]
    • Point Clouds and features from waveform
    • Waveform information improves the semantic
    segmentation task
    (Land Use/Land Cover classification)
    n2D image[Zorzi et al. (2019)]
    • Waveform and height information converted to 2D.
    • Deep Learning(CNN) based methods can easily
    deal with 2D images.
    • Spatial feature extraction improves semantic
    segmentation.
    4
    Direct and spatial feature extraction method
    without converting to feature or grid is needed

    View full-size slide

  5. Sem. Seg. for FW LiDAR Data
    nWaveform or Geometric information
    • Local Group
    ⁃ Waveform improves the segmentation
    performance.
    ‣ e.g. Vegetations have many return peaks.
    • Global Group
    ⁃ An Large area of spatial/geometric context
    improves the segmentation performance.
    ‣ e.g. Roads have a uniform distribution
    of similar waveforms.
    OBJECTIVE: We propose a deep learning-based
    method combining local and global feature
    extractions for FW data sem. seg.
    5
    Images are cited from
    Awange J., Kiema J. (2019) Light Detection And Ranging (LiDAR). In: Environmental Geoinformatics. Environmental Science and Engineering. Springer, Cham. https://doi.org/10.1007/978-3-030-03017-9_21

    View full-size slide

  6. 2. Proposed Method
    6

    View full-size slide

  7. Problem Definition
    nSemantic segmentation task for FW data
    • Input: point (x, y, z) and waveform
    ⁃ ∈ ℝ!×#, $ = $, $, $, $ , $ ∈ ℝ#
    ‣ K: #points, M: x, y, z and length of waveform
    • Output: segmentation result
    ⁃ ∈ ℝ!×&, $ = $,', $,(, ⋯ , $,& , $ ∈ ℝ&
    ‣ K: #points, C: #classes
    7
    Our model predict segmentation results
    from input points and waveforms.
    Input Output

    View full-size slide

  8. Network Architecture
    nLocal Module: Simple 1D CNN
    8
    Class prediction from local waveform information.
    Simple weight shared 1D CNN

    View full-size slide

  9. Network Architecture
    nGlobal Module: PointNet++ based model
    9
    Hierarchical subsample and upsample strategy to
    extract global geometric context information.
    Subsample
    step Upsample
    step

    View full-size slide

  10. Global Module
    nSubsample
    • Chose the typical
    points by subsample.
    nDefine neighbors
    • Convolution needs
    neighbor points.
    10
    Providing waveform relationships in large area.
    nUpsample
    • Recovering the number
    of points from subsampled
    points.
    • Skip connection provides
    vanished high res. info..

    View full-size slide

  11. Convolutional Operation
    nConvolution for waveform
    • Weight shared 1D CNN
    for grouped waveform
    or features.
    • Waveform is defined as
    sequential data like
    audio signals.
    • 1D CNN are widely used
    in audio recognition.
    • MaxPool operation
    after convolution
    are used the same as
    PointNet++.
    11
    Calculating waveform features for grouped data.

    View full-size slide

  12. Optimization
    nLoss function
    !"!#$
    = %&'
    '$"(#$ + %&'
    $")#$
    !"#
    = − $
    $
    %
    $
    &
    '
    &
    ∗ $,&
    log $,&
    )
    = 1/
    )
    log + ∑)*+
    , )
    nTraining detail
    • Optimizer: ADAM with learning rate 0.005
    • Convolution
    ⁃ Radius ball=3, 5, and 15
    ⁃ #neighbors = 16, 32, and 64
    ⁃ #features = 256, 512, and 1024
    • Hardware: P100 on TSUBAME3.0
    12
    Loss for Local module and global module
    at same time
    each loss is weighted cross entropy
    class ratio aware weight

    View full-size slide

  13. Prediction from local and global module
    nCombine outputs from local and global
    13
    The class which waveform improves use local module
    The class which geometry improves use global module

    View full-size slide

  14. 3. Experimental Results
    14

    View full-size slide

  15. Dataset
    nUsed data
    • FW data published by Zorzi et.al,2019
    ⁃ Test: red area
    • Class
    ⁃ Ground, Build., Veg., Power Line, Trans. Tower, and Street Path
    nDivided into 2 group
    • Global group
    ⁃ Geometry is effective.
    ⁃ Ground, Build., Street Path
    • Local group
    ⁃ Local features are effective.
    ⁃ Veg., PowerLine, Trans.Tower
    15
    600 m
    70 m
    24 m
    test
    [Zorzi et al., 2019]

    View full-size slide

  16. Qualitative Evaluation
    16
    As an overall trend, our model predicted correct classes.
    Border area and street path are misclassified.
    Boundary area
    Street Path
    Boundary area

    View full-size slide

  17. Quantitative Evaluation
    17
    Our method achieved the highest f1 scores.
    nUsed data
    • Previous studies
    ⁃ Our model achieved higher scores.
    • Comparison with other deep learning methods
    ⁃ Our local and global strategy achieved higher scores.

    View full-size slide

  18. Ablation Study
    nComparisons of PointNet++ based baselines
    • Waveform
    ⁃ PointNet++ w/ waveform vs. PointNet++ w/o waveform
    • Hierarchical feature extraction
    ⁃ PointNet++ w/ Hierarchical vs. PointNet++ w/o Hierarchical
    • Local module
    ⁃ PointNet++ w/ local vs. PointNet++ w/o local
    18
    We showed the effetiveness of PointNet++ based
    model, waveform information, and local module

    View full-size slide

  19. The Effect of Local Module
    nAcceptance rate from local module
    • Prediction step
    ⁃ Rule-based predictions using
    the output of local module and global module.
    ⁃ Local Module predicts local group (Vegetation,
    Power Line, Transmission Tower), which we
    made the assumption that the local waveform
    information is effective.
    19
    Our local module was able to predict local group

    View full-size slide

  20. 4. Conclusion
    20

    View full-size slide

  21. Conclusion and Future Study
    nConclusion
    • We have proposed a deep learning-based semantic
    segmentation model for full-waveform LiDAR data.
    • Our model consists of the local module which
    predicts class from each waveform, and the global
    module which predicts class from geometry.
    • Experimental results showed that our model
    predicts higher accuracy than previous methods.
    nUnsolved Problems and Future Works
    • Explicit Geometric information
    ⁃ Combine geometric and waveform feature extractions
    • Heuristic combination method
    ⁃ Changing heuristic rule to learnable functions
    21

    View full-size slide