datadrink_09092021_Commission_europeenne

Transcript

1. Multimodal Crop Type Classification fusing Multi-spectral Satellite Images with Farmers Crop Rotations and Local Crop Distribution Valentin Barriere, Martin Claverie European Commission’s Joint Research Center Datadrink, 09/09/21
2. Intro I 1
3. Intro II 2
4. Intro III 3
5. Dataset and Features Crop Types 386 Crop types over 12 years. Aggregated in 36 or 12 general classes: Beetroot, Consumption potatoes, Maize, Seed potatoes, Sillage maize, Spring wheat, Starch potatoes Summer barley Winter barley, Winter wheat, Grassland and Other crops. Sentinel2 EO-based features Time-series of B4 (red band) Surface Reflectance, b8a (near infrared band) Surface Reflectance, Leaf Area Index and Fraction of Absorbed Photosynthetically Active Radiation Local features: Crop Distribution 386-dimension vector representing the distribution of the crops in the surrounding of the parcel (10km radius) The study is focused on data acquired over The Netherlands, for a total of 596,480 parcels 4
6. Language Model Audio Speech Recognition I We modeled our problem as an Audio Speech Recognition (ASR) task: The crop types were modeled as words like in a language model: P(ct+1|ct, ..., c1 ) 5
7. Language Model Audio Speech Recognition II We modeled our problem as an Audio Speech Recognition (ASR) task: The satellite signals were modeled as an acoustic signal, using a sliding window and statistical functionals to temporally aggregate the signal Sliding window of size 30 days, with a step of 15 days 6
8. Language Model Audio Speech Recognition III We modeled our problem as an Audio Speech Recognition (ASR) task: The crop distribution acted as a speaker-specific vocabulary distribution: 7
9. Experiments We used a hierarchical LSTM coded in Pytorch: • We modeled the acoustic signal with a bidirectionnal LSTM with an attention Layer • We modeled the crops with an embedding layer • We fused the acoustic and the crop embeddings using fully connected layers • We modeled the multimodal crop-acoustic vector sequences with a LSTM • We added the crops distribution at the end (high-level information) and fused the modalities using fully connected layers We trained our networks as for a sequence classification task, 2019 as development set and 2020 as test set. 8
10. Results – End of season classification Labels #Modal. 386-class 12-class 10-class Model P R F1 Acc P R F1 Acc P R F1 m-F1 LSTMCrop 1 (C) 28.1 22.4 23.1 73.3 53.1 44.7 43.8 77.2 46.7 39.2 37.1 52.1 LSTMYI [1] 1 (RS) 14.3 9.8 9.9 72.5 75.0 66.3 67.7 90.4 72.1 62.2 63.7 80.3 LSTMRS 1 (RS) 13.7 11.8 10.8 72.5 70.6 69.4 65.3 89.0 67.3 65.3 60.7 76.0 HierbiLSTMRS 1 (RS) 10.7 10.0 9.0 78.7 72.7 71.0 67.7 90.7 69.3 66.6 63.0 79.1 LSTMMM 2 (RS+C) 32.5 26.1 26.2 86.8 79.9 78.5 78.2 93.2 77.9 75.2 75.4 85.1 HierbiLSTMMM 2 (RS+C) 42.0 33.8 35.1 88.5 84.1 80.8 81.3 94.5 82.3 77.9 78.8 87.8 Final 3 (All) 41.0 33.3 34.3 89.7 85.5 81.2 82.6 94.8 84.1 78.3 80.2 88.7 LSTMYI is a model using RNN at the level of the year 9
11. In-season Classification Obtained with a model not trained at all for in-season classification. Good performances already in August. 10
12. Conclusion and Future Works Conclusion: • We studied the integration of crop rotations in a EO-based model and modeled it like an ASR • We added local information using a crop distribution vector • Our method outperforms by a great margin the classical state-of-the-art using only a RNN or a Transformer to model the EO data at the level of a year (LSTMYI ). Future works: • Investigate geographically the errors of our model • Use multimodal aligned or non-aligned time-series fusion models in order to better handle the multimodal aspect • Domain Adaptation using few-shot learning ? 11
13. Questions? 11
14. References i M. Rußwurm, S. Lef` evre, and M. K¨ orner. Breizhcrops: a satellite time series dataset for crop type identification. Time Series Workshop of the 36th ICML, 2019.