Upgrade to Pro — share decks privately, control downloads, hide ads and more …

datadrink_09092021_Commission_europeenne

etalab-ia
September 09, 2021
160

 datadrink_09092021_Commission_europeenne

etalab-ia

September 09, 2021
Tweet

Transcript

  1. Multimodal Crop Type Classification fusing Multi-spectral Satellite Images with Farmers

    Crop Rotations and Local Crop Distribution Valentin Barriere, Martin Claverie European Commission’s Joint Research Center Datadrink, 09/09/21
  2. Dataset and Features Crop Types 386 Crop types over 12

    years. Aggregated in 36 or 12 general classes: Beetroot, Consumption potatoes, Maize, Seed potatoes, Sillage maize, Spring wheat, Starch potatoes Summer barley Winter barley, Winter wheat, Grassland and Other crops. Sentinel2 EO-based features Time-series of B4 (red band) Surface Reflectance, b8a (near infrared band) Surface Reflectance, Leaf Area Index and Fraction of Absorbed Photosynthetically Active Radiation Local features: Crop Distribution 386-dimension vector representing the distribution of the crops in the surrounding of the parcel (10km radius) The study is focused on data acquired over The Netherlands, for a total of 596,480 parcels 4
  3. Language Model Audio Speech Recognition I We modeled our problem

    as an Audio Speech Recognition (ASR) task: The crop types were modeled as words like in a language model: P(ct+1|ct, ..., c1 ) 5
  4. Language Model Audio Speech Recognition II We modeled our problem

    as an Audio Speech Recognition (ASR) task: The satellite signals were modeled as an acoustic signal, using a sliding window and statistical functionals to temporally aggregate the signal Sliding window of size 30 days, with a step of 15 days 6
  5. Language Model Audio Speech Recognition III We modeled our problem

    as an Audio Speech Recognition (ASR) task: The crop distribution acted as a speaker-specific vocabulary distribution: 7
  6. Experiments We used a hierarchical LSTM coded in Pytorch: •

    We modeled the acoustic signal with a bidirectionnal LSTM with an attention Layer • We modeled the crops with an embedding layer • We fused the acoustic and the crop embeddings using fully connected layers • We modeled the multimodal crop-acoustic vector sequences with a LSTM • We added the crops distribution at the end (high-level information) and fused the modalities using fully connected layers We trained our networks as for a sequence classification task, 2019 as development set and 2020 as test set. 8
  7. Results – End of season classification Labels #Modal. 386-class 12-class

    10-class Model P R F1 Acc P R F1 Acc P R F1 m-F1 LSTMCrop 1 (C) 28.1 22.4 23.1 73.3 53.1 44.7 43.8 77.2 46.7 39.2 37.1 52.1 LSTMYI [1] 1 (RS) 14.3 9.8 9.9 72.5 75.0 66.3 67.7 90.4 72.1 62.2 63.7 80.3 LSTMRS 1 (RS) 13.7 11.8 10.8 72.5 70.6 69.4 65.3 89.0 67.3 65.3 60.7 76.0 HierbiLSTMRS 1 (RS) 10.7 10.0 9.0 78.7 72.7 71.0 67.7 90.7 69.3 66.6 63.0 79.1 LSTMMM 2 (RS+C) 32.5 26.1 26.2 86.8 79.9 78.5 78.2 93.2 77.9 75.2 75.4 85.1 HierbiLSTMMM 2 (RS+C) 42.0 33.8 35.1 88.5 84.1 80.8 81.3 94.5 82.3 77.9 78.8 87.8 Final 3 (All) 41.0 33.3 34.3 89.7 85.5 81.2 82.6 94.8 84.1 78.3 80.2 88.7 LSTMYI is a model using RNN at the level of the year 9
  8. In-season Classification Obtained with a model not trained at all

    for in-season classification. Good performances already in August. 10
  9. Conclusion and Future Works Conclusion: • We studied the integration

    of crop rotations in a EO-based model and modeled it like an ASR • We added local information using a crop distribution vector • Our method outperforms by a great margin the classical state-of-the-art using only a RNN or a Transformer to model the EO data at the level of a year (LSTMYI ). Future works: • Investigate geographically the errors of our model • Use multimodal aligned or non-aligned time-series fusion models in order to better handle the multimodal aspect • Domain Adaptation using few-shot learning ? 11
  10. References i M. Rußwurm, S. Lef` evre, and M. K¨

    orner. Breizhcrops: a satellite time series dataset for crop type identification. Time Series Workshop of the 36th ICML, 2019.