Coffee Talk @ ESA PhiLab

A short introduction presentation for ESA-ESRIN Philab in Frascati near Rome, Italy

Transcript

1. Remote Sensing Technology TUM Department of Civil, Geo and Environmental Engineering Technical University of Munich Convolutional-Recurrent Networks for Multi-temporal Classification Coffee Talk @ ESA ESRIN Φ-lab Marc Rußwurm, supervised by Dr. Marco K¨ orner 23rd May, 2018 M. Rußwurm, M. K¨ orner (TUM) | Convolutional-Recurrent Networks for Multi-temporal Classification | Coffee Talk @ ESA ESRIN Φ-lab | 23rd May, 2018 1 / 14
2. About Me Marc Rußwurm born in raised in Bavaria, Germany. recently finished Master Geodesy & Geoinformation @ TU Munich now started PhD @ TUM/DLR Chair of Remote Sensing Technology member Computer Vision Research Group of my Chair scientific interests multi-temporal EO time series with Recurrent Nets natural language processing → EO M. Rußwurm, M. K¨ orner (TUM) | Convolutional-Recurrent Networks for Multi-temporal Classification | Coffee Talk @ ESA ESRIN Φ-lab | 23rd May, 2018 2 / 14
3. About Me Marc Rußwurm born in raised in Bavaria, Germany. recently finished Master Geodesy & Geoinformation @ TU Munich now started PhD @ TUM/DLR Chair of Remote Sensing Technology member Computer Vision Research Group of my Chair scientific interests multi-temporal EO time series with Recurrent Nets natural language processing → EO M. Rußwurm, M. K¨ orner (TUM) | Convolutional-Recurrent Networks for Multi-temporal Classification | Coffee Talk @ ESA ESRIN Φ-lab | 23rd May, 2018 2 / 14
4. About Me Marc Rußwurm born in raised in Bavaria, Germany. recently finished Master Geodesy & Geoinformation @ TU Munich now started PhD @ TUM/DLR Chair of Remote Sensing Technology member Computer Vision Research Group of my Chair scientific interests multi-temporal EO time series with Recurrent Nets natural language processing → EO M. Rußwurm, M. K¨ orner (TUM) | Convolutional-Recurrent Networks for Multi-temporal Classification | Coffee Talk @ ESA ESRIN Φ-lab | 23rd May, 2018 2 / 14
5. About Me I came here for 2 weeks to: help prepare Frontiers Developments Lab Challenge provide some practical feedback on server infrastructure give some deep learning experience (Docker, Tensorflow, Jupyter, GitHub) tell everyone that GANs and RNNs are great meet nice people and get invited to coffee and get a perspective for my PhD M. Rußwurm, M. K¨ orner (TUM) | Convolutional-Recurrent Networks for Multi-temporal Classification | Coffee Talk @ ESA ESRIN Φ-lab | 23rd May, 2018 3 / 14
6. About Me I came here for 2 weeks to: help prepare Frontiers Developments Lab Challenge provide some practical feedback on server infrastructure give some deep learning experience (Docker, Tensorflow, Jupyter, GitHub) tell everyone that GANs and RNNs are great meet nice people and get invited to coffee and get a perspective for my PhD M. Rußwurm, M. K¨ orner (TUM) | Convolutional-Recurrent Networks for Multi-temporal Classification | Coffee Talk @ ESA ESRIN Φ-lab | 23rd May, 2018 3 / 14
7. Multi-temporal classification spectral features + spatial features M. Rußwurm, M. K¨ orner (TUM) | Convolutional-Recurrent Networks for Multi-temporal Classification | Coffee Talk @ ESA ESRIN Φ-lab | 23rd May, 2018 4 / 14
8. Multi-temporal classification spectral features + spatial features + temporal features data is available in 10–30 m resolution with acquisition period of 5–10 d + free of charge. so why dont we use time series all the time? M. Rußwurm, M. K¨ orner (TUM) | Convolutional-Recurrent Networks for Multi-temporal Classification | Coffee Talk @ ESA ESRIN Φ-lab | 23rd May, 2018 4 / 14
9. Multi-temporal classification spectral features + spatial features + temporal features data is available in 10–30 m resolution with acquisition period of 5–10 d + free of charge. so why dont we use time series all the time? M. Rußwurm, M. K¨ orner (TUM) | Convolutional-Recurrent Networks for Multi-temporal Classification | Coffee Talk @ ESA ESRIN Φ-lab | 23rd May, 2018 4 / 14
10. Multi-temporal classification spectral features + spatial features + temporal features data is available in 10–30 m resolution with acquisition period of 5–10 d + free of charge. so why dont we use time series all the time? M. Rußwurm, M. K¨ orner (TUM) | Convolutional-Recurrent Networks for Multi-temporal Classification | Coffee Talk @ ESA ESRIN Φ-lab | 23rd May, 2018 4 / 14
11. Multi-temporal classification Because it’s a pain to find a functional relation y = f(x0 , x1 , . . . , xT ) between 4-D multi-temporal reflections x ∈ {x0 , x1 , . . . , xT }, xt ∈ Rh×w×d and the desired output y ∈ Rh×w×n M. Rußwurm, M. K¨ orner (TUM) | Convolutional-Recurrent Networks for Multi-temporal Classification | Coffee Talk @ ESA ESRIN Φ-lab | 23rd May, 2018 5 / 14
12. Multi-temporal classification Because it’s a pain to find a functional relation y = f(x0 , x1 , . . . , xT ) between 4-D multi-temporal reflections x ∈ {x0 , x1 , . . . , xT }, xt ∈ Rh×w×d and the desired output y ∈ Rh×w×n M. Rußwurm, M. K¨ orner (TUM) | Convolutional-Recurrent Networks for Multi-temporal Classification | Coffee Talk @ ESA ESRIN Φ-lab | 23rd May, 2018 5 / 14
13. How do we solve this? = f( 0 , 1 , . . . , T ) model-driven define the functional model with domain-specific expert knowledge break the model into solvable sub-tasks i.e. preproc. → feat. extraction → classification less data required model fixed by expert, optimize parameters better guide-able by RS expert × data dimensionality reduced information lost? × plenty of model assumptions data-driven learn an approximated functional model from provided data expert designs a network structure with sufficient capacity × a lot of data required learn model and parameters × non-deterministic training all available data can be utilized network broadly applicable only restricted by IO format M. Rußwurm, M. K¨ orner (TUM) | Convolutional-Recurrent Networks for Multi-temporal Classification | Coffee Talk @ ESA ESRIN Φ-lab | 23rd May, 2018 6 / 14
14. How do we solve this? = f( 0 , 1 , . . . , T ) model-driven define the functional model with domain-specific expert knowledge break the model into solvable sub-tasks i.e. preproc. → feat. extraction → classification less data required model fixed by expert, optimize parameters better guide-able by RS expert × data dimensionality reduced information lost? × plenty of model assumptions data-driven learn an approximated functional model from provided data expert designs a network structure with sufficient capacity × a lot of data required learn model and parameters × non-deterministic training all available data can be utilized network broadly applicable only restricted by IO format M. Rußwurm, M. K¨ orner (TUM) | Convolutional-Recurrent Networks for Multi-temporal Classification | Coffee Talk @ ESA ESRIN Φ-lab | 23rd May, 2018 6 / 14
15. How do we solve this? = f( 0 , 1 , . . . , T ) model-driven define the functional model with domain-specific expert knowledge break the model into solvable sub-tasks i.e. preproc. → feat. extraction → classification less data required model fixed by expert, optimize parameters better guide-able by RS expert × data dimensionality reduced information lost? × plenty of model assumptions data-driven learn an approximated functional model from provided data expert designs a network structure with sufficient capacity × a lot of data required learn model and parameters × non-deterministic training all available data can be utilized network broadly applicable only restricted by IO format M. Rußwurm, M. K¨ orner (TUM) | Convolutional-Recurrent Networks for Multi-temporal Classification | Coffee Talk @ ESA ESRIN Φ-lab | 23rd May, 2018 6 / 14
16. How do we solve this? = f( 0 , 1 , . . . , T ) model-driven define the functional model with domain-specific expert knowledge break the model into solvable sub-tasks i.e. preproc. → feat. extraction → classification less data required model fixed by expert, optimize parameters better guide-able by RS expert × data dimensionality reduced information lost? × plenty of model assumptions data-driven learn an approximated functional model from provided data expert designs a network structure with sufficient capacity × a lot of data required learn model and parameters × non-deterministic training all available data can be utilized network broadly applicable only restricted by IO format M. Rußwurm, M. K¨ orner (TUM) | Convolutional-Recurrent Networks for Multi-temporal Classification | Coffee Talk @ ESA ESRIN Φ-lab | 23rd May, 2018 6 / 14
17. Our Agenda pure data-driven end-to-end learning no pre/postprocessing of data top-of-atmosphere data no atmospheric correction no cloud filtering no additional image registration x end-to-end learning y M. Rußwurm, M. K¨ orner (TUM) | Convolutional-Recurrent Networks for Multi-temporal Classification | Coffee Talk @ ESA ESRIN Φ-lab | 23rd May, 2018 7 / 14
18. Our Agenda pure data-driven end-to-end learning no pre/postprocessing of data top-of-atmosphere data no atmospheric correction no cloud filtering no additional image registration x end-to-end learning y M. Rußwurm, M. K¨ orner (TUM) | Convolutional-Recurrent Networks for Multi-temporal Classification | Coffee Talk @ ESA ESRIN Φ-lab | 23rd May, 2018 7 / 14
19. Our Agenda pure data-driven end-to-end learning no pre/postprocessing of data top-of-atmosphere data no atmospheric correction no cloud filtering no additional image registration x end-to-end learning y M. Rußwurm, M. K¨ orner (TUM) | Convolutional-Recurrent Networks for Multi-temporal Classification | Coffee Talk @ ESA ESRIN Φ-lab | 23rd May, 2018 7 / 14
20. Our Agenda pure data-driven end-to-end learning no pre/postprocessing of data top-of-atmosphere data no atmospheric correction no cloud filtering no additional image registration x end-to-end learning y M. Rußwurm, M. K¨ orner (TUM) | Convolutional-Recurrent Networks for Multi-temporal Classification | Coffee Talk @ ESA ESRIN Φ-lab | 23rd May, 2018 7 / 14
21. Our Agenda pure data-driven end-to-end learning no pre/postprocessing of data top-of-atmosphere data no atmospheric correction no cloud filtering no additional image registration x end-to-end learning y M. Rußwurm, M. K¨ orner (TUM) | Convolutional-Recurrent Networks for Multi-temporal Classification | Coffee Talk @ ESA ESRIN Φ-lab | 23rd May, 2018 7 / 14
22. Our Agenda pure data-driven end-to-end learning no pre/postprocessing of data top-of-atmosphere data no atmospheric correction no cloud filtering no additional image registration x end-to-end learning y M. Rußwurm, M. K¨ orner (TUM) | Convolutional-Recurrent Networks for Multi-temporal Classification | Coffee Talk @ ESA ESRIN Φ-lab | 23rd May, 2018 7 / 14
23. Sequence-to-Sequence Translation Recurrent encoder-decoder nets used for sequential tasks, such as natural language processing input sequence representation output sequence It’s raining in ESRIN Es regnet in ESRIN representation cT 0 0 0 M. Rußwurm, M. K¨ orner (TUM) | Convolutional-Recurrent Networks for Multi-temporal Classification | Coffee Talk @ ESA ESRIN Φ-lab | 23rd May, 2018 8 / 14
24. Bidirectional Convolutional-Recurrent Network jan → dec x0 x1 . . . xT 0 0 cfw T M. Rußwurm, M. K¨ orner (TUM) | Convolutional-Recurrent Networks for Multi-temporal Classification | Coffee Talk @ ESA ESRIN Φ-lab | 23rd May, 2018 9 / 14
25. Bidirectional Convolutional-Recurrent Network jan → dec x0 x1 . . . xT 0 0 cfw T dec → jan xT xT −1 . . . x0 0 0 cbw T M. Rußwurm, M. K¨ orner (TUM) | Convolutional-Recurrent Networks for Multi-temporal Classification | Coffee Talk @ ESA ESRIN Φ-lab | 23rd May, 2018 9 / 14
26. Bidirectional Convolutional-Recurrent Network jan → dec x0 x1 . . . xT 0 0 cfw T dec → jan xT xT −1 . . . x0 0 0 cbw T classification prediction Ground Truth argmax H(y, ˆ y) M. Rußwurm, M. K¨ orner (TUM) | Convolutional-Recurrent Networks for Multi-temporal Classification | Coffee Talk @ ESA ESRIN Φ-lab | 23rd May, 2018 9 / 14
27. Qualitative Examples xRGB,t labels y pred. ˆ y H(y, ˆ y) activation activation activation activation maize meadow peas rape 1 0 spelt wheat s.barley maize 1 0 meadow wheat potato maize 1 0 aspar. bean hop maize mead. peas pot. rape soyb. beet s.barl oat w.barl rye spelt tritic wheat M. Rußwurm, M. K¨ orner (TUM) | Convolutional-Recurrent Networks for Multi-temporal Classification | Coffee Talk @ ESA ESRIN Φ-lab | 23rd May, 2018 10 / 14
28. Qualitative Examples xRGB,t labels y pred. ˆ y H(y, ˆ y) activation activation activation activation rye triticale s.barley w.barley 1 0 rye wheat triticale s.barley 1 0 wheat meadow maize w.barley 1 0 aspar. bean hop maize mead. peas pot. rape soyb. beet s.barl oat w.barl rye spelt tritic wheat M. Rußwurm, M. K¨ orner (TUM) | Convolutional-Recurrent Networks for Multi-temporal Classification | Coffee Talk @ ESA ESRIN Φ-lab | 23rd May, 2018 11 / 14
29. Cloud Sensitive Cells LSTM cell 47 of 256 x f(47) i(47) j(47) c(47) t1 t2 . . . . . . . . . . . . . . . t10 t11 t12 t13 t14 t15 . . . . . . . . . . . . . . . t31 t32 t33 t34 t35 0 1 0 1 -1 1 -1 1 M. Rußwurm, M. K¨ orner (TUM) | Convolutional-Recurrent Networks for Multi-temporal Classification | Coffee Talk @ ESA ESRIN Φ-lab | 23rd May, 2018 12 / 14
30. Ablation Experiment on Cloudy Data 03. Jan 13. Jan 20. Jan 21. Jan 28. Jan 12. Feb 11. M¨ ar 20. M¨ ar 23. M¨ ar 03. Apr 13 Apr. 19. Apr 22. Apr 29. Apr 02. Mai 10. Mai 22. Mai 29. Mai 08. Jun 18. Jun 28. Jan 02. Jul 14. Jul 18. Jul 21. Jul 28. Jul 30. Jul 07. Aug 17. Aug 20. Aug 28. Aug 21. Aug 09. Sep. 12. Sep.. 18. Sep. 26. Sep. 29. Sep. 09. Okt. 18. Okt. 28. Okt. 09. Nov. 15. Nov. 18. Nov. 28. Nov. 06. Dez. 08. Dez. 1 % 10 % 25 % 50 % 100 % image acquisition dates cloud coverage 0 2 · 105 4 · 105 6 · 105 8 · 105 0.2 0.4 0.6 0.8 1 samples seen ov. acc. 0 2 · 105 4 · 105 6 · 105 8 · 105 0.85 0.9 0.95 samples seen ov. acc. < 1 % (4 obs.) < 10 % (10 obs.) < 25 % (17 obs.) < 50 % (23 obs.) all images (46 obs.) M. Rußwurm, M. K¨ orner (TUM) | Convolutional-Recurrent Networks for Multi-temporal Classification | Coffee Talk @ ESA ESRIN Φ-lab | 23rd May, 2018 13 / 14
31. Papers & Code Github + DockerHub https://github.com/TUM-LMF/MTLCC http://www.lmf.bgu.tum.de/vision/ Rußwurm, M. and K¨ orner, M. (2017). Temporal Vegetation Modelling using Long Short-Term Memory Networks for Crop Identification from Medium-Resolution Multi-Spectral Satellite Images. In IEEE/ISPRS EarthVision 2017 Workshop, Proceedings of the IEEE CVPR Workshops. Rußwurm M., K¨ orner M. (2018). Multi-Temporal Land Cover Classification with Sequential Recurrent Encoders. ISPRS International Journal of Geo-Information. https://arxiv.org/abs/1802.02080. (in review) M. Rußwurm, M. K¨ orner (TUM) | Convolutional-Recurrent Networks for Multi-temporal Classification | Coffee Talk @ ESA ESRIN Φ-lab | 23rd May, 2018 14 / 14