Do Deep Neural Networks Learn Full Waveform LiDAR Data? Takayuki SHINOHARA Matsuoka Laboratory, Tokyo Institute of Technology 2019/11/18 Joint Student Seminar on Remote Sensing and Geoinformatics

Background and Objective 2

Full Waveform LiDAR nAdvantages • Recording the entire reflected signal “waveform” discretely. • Providing not only 3D point clouds, but also additional information about the target properties. nData Analysis • Costly and time consuming at manual processing. Automatic analysis method for "raw” waveform data is needed. 3 Cited from “Urban land cover classification using airborne LiDAR data: A review”

Raw Waveform analysis nDeep learning (S. Zorzi et al., 2019) • 2 stages supervised classification ‣ 1st stage: Waveform analysis by 1D CNN ⁃ Highly miss classification ‣ 2nd stage: Spatial analysis by 2D CNN ⁃ Raw classification results are converted to grid data ⁃ Prediction the class each grid. • Problems ‣ Spatial raw waveforms are not used in 1st stage. Þ Spatial learning method for raw waveform data are needed to improve performance. 4 Waveform Spatial

Solve problem nPoint Net(Qi et.al, 2017) • One of the deep learning method for spatially irregular data. • Input order invariant network. nAuto Encoder as representation learning • Representation learning is data driven feature extraction method. • Auto Encoder is one of the method. • Auto Encoder can extract low dimensional latent vector from high dimensional data such as Image or some spatial data. Objective in this study: Using a deep learning method, a new representation learning method for spatially distributed raw full-waveform data. 5

Proposed Method 6

Proposed network (FWNetAE) nAuto Encoder for full waveform data • Encoder: PointNet based • Decoder: simple Multi Layer Perceptron(MLP) 7 62 2,048 … 1D Conv features features features features features features … Input full waveform LiDAR Data Output full-waveform LiDAR Data Max Pool MLP 62 2,048 Latent vector PointNet based Encoder Decoder T-nets

Proposed network (FWNetAE) nEncoder: Point Net based Architecture • 1D CNN: Extract local features • MaxPool: Extract grobal features • T-nets: Extract rotation invariant features 8 62 2,048 … 1D Conv features features features features features features … Input full waveform LiDAR Data Output full-waveform LiDAR Data Max Pool MLP 62 2,048 Latent vector PointNet based Encoder Decoder T-nets

Proposed network (FWNetAE) nDecoder: MLP based architecture • Fully connected layers to produce reconstructed data ! of 2,048 × 62 dimensions, i.e., the same as those of input data 9 62 2,048 … 1D Conv features features features features features features … Input full waveform LiDAR Data Output full-waveform LiDAR Data Max Pool MLP 62 2,048 Latent vector PointNet based Encoder Decoder T-nets

Loss function nLoss function for reconstruction • FWNetAE aims at reconstructing target reconstructed data ! . • Reconstruction loss functions. ‣ Spatial matching loss ‣ Waveform reconstruction 10 %&'() = 1 2N . /01 2 3 / − 6 / 7 + / − 6 / 7), 1 <'=)>?@A = 1 2N . /01 2 . B01 C /,B − ̂ /,B 7 . 2 Optimization process Minimize these function

Experimental Results 11

Dataset and Implementation nDataset • Dublin City Dataset ‣ Sample Size: ⁃ Train, Val, Test: 300,000, 100,000 100,000 nImplementation and training results • Implementation ‣ Framework: PyTorch ‣ Hardware: 7th generation 3.8 GHz i7 CPU, 32 GB of RAM, and a Quadro P6000 GPU • Result ‣ Training time: approximately 9 hours ‣ Reconstruction error(test data): spatial=0.051, waveform=0.29 12 Used data

Reconstruction Results 1/2 nSpatial reconstruction 13 A matching shape was observed. 27 Failure case

Reconstruction Results 2/2 nWaveform reconstruction 14 28 Failure case A matching shape was observed.

Latent Space Visualization nComparison of some method 15 Nonspatial AE PCA Proposed method (FWNetAE) Learnable algorithms Not Learnable

Conclusion and Future Study 16

Conclusions and Future Study nConclusions • This paper presents a novel representation learning method for spatially distributed full-waveform data observed from an ALS using an AE-based architecture called FWNetAE. • The results demonstrate a generalization error for invisible test data. • Moreover, the FWNetAE encoded a meaningful latent vector and the decoders reconstructed the spatial geometry and its waveform value from the encoded latent vector. • However, the PointNet-based encoders could not extract features at various resolutions. nFuture Study • Modern Hieratical learning: PointNet++, Dynamic Graph CNN • Application for Supervised Learning 17

Do Deep Neural Networks Learn Full Waveform LiDAR Data? Yes, they do. 18