ERA5 climate reanalysis in Earth Engine

ERA5 climate reanalysis in Earth Engine

This presentation was given for the #EarthEngineVirtualMeetup on Dec 4, 2019. It gives some insights into making 7 TBs of ERA5 climate reananalysis available to Google Earth Engine.

157d70ab1729f9665879a445fb6b7c87?s=128

Julia Wagemann

December 04, 2019
Tweet

Transcript

  1. ERA5 climate reanalysis in Earth Engine Julia Wagemann PhD candidate

    at University of Marburg Visiting Scientist at ECMWF @JuliaWagemann #EarthEngineVirtualMeetup 4 Dec 2019
  2. That’s me Intersection between data provider and data users Making

    open data better accessible to users
  3. 5 PBs temporal resolution atmospheric parameters spatial resolution FULL, FREE

    and OPEN data policy ERA5
  4. Principle components of a Big Earth Data workflow Access Processing

    Visualization
  5. Reproducibility challenge - DATA ACCESSIBILITY • different data are accessible

    via different data access systems • it is still about downloading data • community-specific data formats (GRIB, NetCDF, GeoTiff) • data structure and complexity (analyses vs forecast, multiple dimensions) Access
  6. Islands of Open Big Earth Data

  7. Islands of Open Big Earth Data Meteorological / climate community

    Earth Observation community
  8. Islands of Open Big Earth Data Meteorological / climate community

    Earth Observation community • Copernicus Climate Data Store • GRIB, NetCDF • Google Earth Engine • GeoTiff, JPEG2000
  9. ERA5 in GEE in numbers 7 TBs 104,000 243,113 3

    IMAGE COLLECTIONS 9 VARIABLES HOURLY | DAILY | MONTHLY • 2m air temperature (min, mean, max) • 2m dewpoint temperature • Total precipitation • Surface Pressure • Mean sea-level pressure • 10m u- and v-component of wind
  10. None
  11. Manifest upload in Earth Engine • JSON file • Definition

    of the EE asset • Links to GCP URI • 1 manifest per time stamp ◦ 9 ‘bands’ Upload via CLI earthengine --use_cloud_api upload image --manifest manifest.json
  12. Interoperability of data systems - A CHALLENGE BOTTLENECK DATA DOWNLOAD

    Non-modifiable EE assets
  13. https://developers.google.com/earth-engine/datasets/catalog/ECMWF_ERA5_DAILY https://developers.google.com/earth-engine/datasets/catalog/ECMWF_ERA5_MONTHLY

  14. ERA5 surface pressure

  15. What’s next? • Ingest of hourly files → 359,160 assets

    with 9 bands!!! • Making workflow and python code available on Github (after a clean-up)
  16. Resources • Climate Data Store (https://cds.climate.copernicus.eu/#!/home) • ERA5 climate reanalysis

    (https://cds.climate.copernicus.eu/cdsapp#!/dataset/reanalysis-era5-single-levels?tab=overview) • Python library ‘cdsapi’ (https://pypi.org/project/cdsapi/) • Python library ‘xarray’ (http://xarray.pydata.org/en/stable/index.html) • Google Cloud Platform (https://cloud.google.com/) • google-cloud-storage API (https://cloud.google.com/storage/docs/reference/libraries) • earthengine-api (https://github.com/google/earthengine-api) • Manifest upload (https://developers.google.com/earth-engine/image_manifest)
  17. Thank you! Questions? Julia Wagemann PhD candidate at University of

    Marburg Visiting Scientist at ECMWF @JuliaWagemann #EarthEngineVirtualMeetup 4 Dec 2019