ERA5 climate reanalysis in Earth Engine

This presentation was given for the #EarthEngineVirtualMeetup on Dec 4, 2019. It gives some insights into making 7 TBs of ERA5 climate reananalysis available to Google Earth Engine.

Transcript

1. ERA5 climate
   reanalysis
   in Earth Engine Julia Wagemann
   PhD candidate at University of Marburg
   Visiting Scientist at ECMWF
   @JuliaWagemann
   #EarthEngineVirtualMeetup
   4 Dec 2019
   

   View Slide

2. That’s me
   Intersection between data
   provider and data users
   Making open data better
   accessible to users
   

   View Slide

3. 5 PBs
   temporal
   resolution
   atmospheric
   parameters
   spatial
   resolution
   FULL, FREE and OPEN
   data policy
   ERA5
   

   View Slide

4. Principle components of a Big Earth Data workflow
   Access Processing Visualization
   

   View Slide

5. Reproducibility challenge - DATA ACCESSIBILITY
   ● different data are accessible via different
   data access systems
   ● it is still about downloading data
   ● community-specific data formats
   (GRIB, NetCDF, GeoTiff)
   ● data structure and complexity
   (analyses vs forecast, multiple dimensions)
   Access
   

   View Slide

6. Islands of Open Big Earth Data
   

   View Slide

7. Islands of Open Big Earth Data
   Meteorological /
   climate community
   Earth Observation
   community
   

   View Slide

8. Islands of Open Big Earth Data
   Meteorological /
   climate community
   Earth Observation
   community
   ● Copernicus
   Climate Data
   Store
   ● GRIB, NetCDF
   ● Google Earth
   Engine
   ● GeoTiff,
   JPEG2000
   

   View Slide

9. ERA5 in GEE in numbers
   7 TBs
   104,000
   243,113
   3 IMAGE COLLECTIONS
   9 VARIABLES
   HOURLY | DAILY | MONTHLY
   ● 2m air temperature (min,
   mean, max)
   ● 2m dewpoint temperature
   ● Total precipitation
   ● Surface Pressure
   ● Mean sea-level pressure
   ● 10m u- and v-component
   of wind
   

   View Slide

10. View Slide

11. Manifest upload in Earth Engine
    ● JSON file
    ● Definition of the EE asset
    ● Links to GCP URI
    ● 1 manifest per time stamp
    ○ 9 ‘bands’
    Upload via CLI
    earthengine --use_cloud_api upload image
    --manifest manifest.json
    

    View Slide

12. Interoperability of data systems - A CHALLENGE
    BOTTLENECK
    DATA DOWNLOAD
    Non-modifiable EE
    assets
    

    View Slide

13. https://developers.google.com/earth-engine/datasets/catalog/ECMWF_ERA5_DAILY
    https://developers.google.com/earth-engine/datasets/catalog/ECMWF_ERA5_MONTHLY
    

    View Slide

14. ERA5 surface pressure
    

    View Slide

15. What’s
    next?
    ● Ingest of hourly files →
    359,160 assets with 9 bands!!!
    ● Making workflow and python
    code available on Github (after
    a clean-up)
    

    View Slide

16. Resources
    ● Climate Data Store (https://cds.climate.copernicus.eu/#!/home)
    ● ERA5 climate reanalysis
    (https://cds.climate.copernicus.eu/cdsapp#!/dataset/reanalysis-era5-single-levels?tab=overview)
    ● Python library ‘cdsapi’ (https://pypi.org/project/cdsapi/)
    ● Python library ‘xarray’ (http://xarray.pydata.org/en/stable/index.html)
    ● Google Cloud Platform (https://cloud.google.com/)
    ● google-cloud-storage API (https://cloud.google.com/storage/docs/reference/libraries)
    ● earthengine-api (https://github.com/google/earthengine-api)
    ● Manifest upload (https://developers.google.com/earth-engine/image_manifest)
    

    View Slide

17. Thank you!
    Questions?
    Julia Wagemann
    PhD candidate at University of Marburg
    Visiting Scientist at ECMWF
    @JuliaWagemann
    #EarthEngineVirtualMeetup
    4 Dec 2019
    

    View Slide