ERA5 climate reanalysis in Earth Engine

This presentation was given for the #EarthEngineVirtualMeetup on Dec 4, 2019. It gives some insights into making 7 TBs of ERA5 climate reananalysis available to Google Earth Engine.

1. ERA5 climate reanalysis in Earth Engine Julia Wagemann PhD candidate at University of Marburg Visiting Scientist at ECMWF @JuliaWagemann #EarthEngineVirtualMeetup 4 Dec 2019
2. That’s me Intersection between data provider and data users Making open data better accessible to users
3. 5 PBs temporal resolution atmospheric parameters spatial resolution FULL, FREE and OPEN data policy ERA5
4. Principle components of a Big Earth Data workflow Access Processing Visualization
5. Reproducibility challenge - DATA ACCESSIBILITY • different data are accessible via different data access systems • it is still about downloading data • community-specific data formats (GRIB, NetCDF, GeoTiff) • data structure and complexity (analyses vs forecast, multiple dimensions) Access
6. Islands of Open Big Earth Data
7. Islands of Open Big Earth Data Meteorological / climate community Earth Observation community
8. Islands of Open Big Earth Data Meteorological / climate community Earth Observation community • Copernicus Climate Data Store • GRIB, NetCDF • Google Earth Engine • GeoTiff, JPEG2000
9. ERA5 in GEE in numbers 7 TBs 104,000 243,113 3 IMAGE COLLECTIONS 9 VARIABLES HOURLY | DAILY | MONTHLY • 2m air temperature (min, mean, max) • 2m dewpoint temperature • Total precipitation • Surface Pressure • Mean sea-level pressure • 10m u- and v-component of wind
11. Manifest upload in Earth Engine • JSON file • Definition of the EE asset • Links to GCP URI • 1 manifest per time stamp ◦ 9 ‘bands’ Upload via CLI earthengine --use_cloud_api upload image --manifest manifest.json
12. Interoperability of data systems - A CHALLENGE BOTTLENECK DATA DOWNLOAD Non-modifiable EE assets
13. https://developers.google.com/earth-engine/datasets/catalog/ECMWF_ERA5_DAILY https://developers.google.com/earth-engine/datasets/catalog/ECMWF_ERA5_MONTHLY
14. ERA5 surface pressure
15. What’s next? • Ingest of hourly files → 359,160 assets with 9 bands!!! • Making workflow and python code available on Github (after a clean-up)
16. Resources • Climate Data Store (https://cds.climate.copernicus.eu/#!/home) • ERA5 climate reanalysis (https://cds.climate.copernicus.eu/cdsapp#!/dataset/reanalysis-era5-single-levels?tab=overview) • Python library ‘cdsapi’ (https://pypi.org/project/cdsapi/) • Python library ‘xarray’ (http://xarray.pydata.org/en/stable/index.html) • Google Cloud Platform (https://cloud.google.com/) • google-cloud-storage API (https://cloud.google.com/storage/docs/reference/libraries) • earthengine-api (https://github.com/google/earthengine-api) • Manifest upload (https://developers.google.com/earth-engine/image_manifest)
17. Thank you! Questions? Julia Wagemann PhD candidate at University of Marburg Visiting Scientist at ECMWF @JuliaWagemann #EarthEngineVirtualMeetup 4 Dec 2019