Upgrade to Pro — share decks privately, control downloads, hide ads and more …

ERA5 climate reanalysis in Earth Engine

ERA5 climate reanalysis in Earth Engine

This presentation was given for the #EarthEngineVirtualMeetup on Dec 4, 2019. It gives some insights into making 7 TBs of ERA5 climate reananalysis available to Google Earth Engine.

157d70ab1729f9665879a445fb6b7c87?s=128

Julia Wagemann

December 04, 2019
Tweet

More Decks by Julia Wagemann

Other Decks in Technology

Transcript

  1. ERA5 climate reanalysis in Earth Engine Julia Wagemann PhD candidate

    at University of Marburg Visiting Scientist at ECMWF @JuliaWagemann #EarthEngineVirtualMeetup 4 Dec 2019
  2. That’s me Intersection between data provider and data users Making

    open data better accessible to users
  3. 5 PBs temporal resolution atmospheric parameters spatial resolution FULL, FREE

    and OPEN data policy ERA5
  4. Principle components of a Big Earth Data workflow Access Processing

    Visualization
  5. Reproducibility challenge - DATA ACCESSIBILITY • different data are accessible

    via different data access systems • it is still about downloading data • community-specific data formats (GRIB, NetCDF, GeoTiff) • data structure and complexity (analyses vs forecast, multiple dimensions) Access
  6. Islands of Open Big Earth Data

  7. Islands of Open Big Earth Data Meteorological / climate community

    Earth Observation community
  8. Islands of Open Big Earth Data Meteorological / climate community

    Earth Observation community • Copernicus Climate Data Store • GRIB, NetCDF • Google Earth Engine • GeoTiff, JPEG2000
  9. ERA5 in GEE in numbers 7 TBs 104,000 243,113 3

    IMAGE COLLECTIONS 9 VARIABLES HOURLY | DAILY | MONTHLY • 2m air temperature (min, mean, max) • 2m dewpoint temperature • Total precipitation • Surface Pressure • Mean sea-level pressure • 10m u- and v-component of wind
  10. None
  11. Manifest upload in Earth Engine • JSON file • Definition

    of the EE asset • Links to GCP URI • 1 manifest per time stamp ◦ 9 ‘bands’ Upload via CLI earthengine --use_cloud_api upload image --manifest manifest.json
  12. Interoperability of data systems - A CHALLENGE BOTTLENECK DATA DOWNLOAD

    Non-modifiable EE assets
  13. https://developers.google.com/earth-engine/datasets/catalog/ECMWF_ERA5_DAILY https://developers.google.com/earth-engine/datasets/catalog/ECMWF_ERA5_MONTHLY

  14. ERA5 surface pressure

  15. What’s next? • Ingest of hourly files → 359,160 assets

    with 9 bands!!! • Making workflow and python code available on Github (after a clean-up)
  16. Resources • Climate Data Store (https://cds.climate.copernicus.eu/#!/home) • ERA5 climate reanalysis

    (https://cds.climate.copernicus.eu/cdsapp#!/dataset/reanalysis-era5-single-levels?tab=overview) • Python library ‘cdsapi’ (https://pypi.org/project/cdsapi/) • Python library ‘xarray’ (http://xarray.pydata.org/en/stable/index.html) • Google Cloud Platform (https://cloud.google.com/) • google-cloud-storage API (https://cloud.google.com/storage/docs/reference/libraries) • earthengine-api (https://github.com/google/earthengine-api) • Manifest upload (https://developers.google.com/earth-engine/image_manifest)
  17. Thank you! Questions? Julia Wagemann PhD candidate at University of

    Marburg Visiting Scientist at ECMWF @JuliaWagemann #EarthEngineVirtualMeetup 4 Dec 2019