Open Meteorological and Climate Data - Building bridges between user communities

This presentation was given for a keynote at the FOSS4G 2019 conference in Bucharest, Romania on 29 August 2019.

The talk gives an overview of freely available environmental datasets from the European Centre for Medium-Range Weather Forecasts as part of the Copernicus programme. It covers aspects of open data and the importance of making them not just open, but also accessible and useable. The talk highlights the perspective of a data user trying to work with large volumes of meteorological and climate data.

Transcript

1. Open METEOROLOGICAL and CLIMATE DATA Julia Wagemann Geospatial data consultant, Visiting Scientist @ECMWF, PhD candidate @MarbugUniversity @JuliaWagemann jwagemann.github.io Building BRIDGES between user communities!
3. CLIMATE ERA5 reanalysis • 1979 - almost Near-Real-Time ◦ (soon from 1950 onwards) • Temporal resolution: hourly • Spatial resolution: ~ 30 km (ERA5-Land: 9 km) Seasonal forecasts • monthly forecasts up to 6 months ahead • Spatial resolution: 10 km
4. CAMS reanalysis • 2003 - 2016 (will be extended • Spatial resolution: 80 km CAMS Forecasts • 3-hourly forecasts up to 5 days in advance • Spatial resolution: 80 km Parameters: Ozone, CO2, SO2, NO2, ... AIR QUALITY
5. Fire danger indices • Fire-Weather Index • Fine Fuel Moisture Code • Duff Moisture Code • Drought Code • Build-Up Index • Initial Spread Index • Daily Severity Rating Time period: 1 Jan 1980 - 31 Dec 2018 Temporal resolution: daily Spatial resolution: 80 km FIRE DANGER
6. FLOOD River discharge • GloFAS 30-day forecast ◦ 1 Jan 2018 - realtime ◦ Temporal resolution: daily ◦ Spatial resolution: 10 km • GloFAS 30-day reanalysis ◦ 1981-2017 ◦ Temporal resolution: daily ◦ Spatial resolution: 10 km
7. Copernicus FULL, FREE and OPEN data policy
8. *Aspects like FORMAT, STRUCTURE, MACHINE-READABILITY make the data MORE USABLE But these DO NOT make the data MORE OPEN! *European Data Portal (2019): elearning module #1 - What is ‘open’ data?
9. Do aspects of non-interoperability, data complexity, community-specific formats, … make the data LESS OPEN? u
10. CLIMATE OGC WCS Array database
11. BIG DATA is NOT A NEW TERM 1
12. OPERATIONAL means ‘really’ OPERATIONAL 2 24 7 /
13. We talk about FIELDS and GRID POINTS not bands and pixels 3
14. STANDARDS and INTEROPERABILITY are NO NEW TERMS 4
15. 5 We like CUSTOM-BUILT SOFTWARE developed in-house
16. #1 Trend* *PwC for European Commission (2019): Copernicus market report (EO) expert user End user Intermediate user Non EO expert DIVERSIFICATION of users and their demands Decision-maker Policy maker Geospatial data user
17. 1 DATA COMPLEXITY CHALLENGE ENSEMBLES FORECASTS PRESSURE LEVELS
18. User requirements survey Wagemann et al. (2019): in preparation 1 2 3 4
19. How much of the ‘OPEN’ DATA, that is produced, is USED? 11 TBs forecast data per day 10 TBs data from the Sentinels per day 5 PBs of ERA5 data
20. 1 2 Programmatic access Web Portal 3 FTP on request 1 CDS Toolbox Download On-demand 4 Zenodo 2 WMS
21. MIRROR ARCHIVES as BRIDGES between different user communities Earth Observation community ERA5 data
22. Example of NON-INTEROPERABILITY of systems: ERA5 in GEE Download Convert Upload Ingest SOON: 7 ERA5 parameters as part of GEE public data catalogue
23. 1 INTEROPERABILITY and STANDARD PROTOCOLS NEED WMS WCS WPS
24. 2 DRIVERS and PACKAGES for PYTHON and R NEED cfgrib 0.9.7.1 Magics 1.0.7 eccodes-python 0.9.2 metview 1.2.0 cdsapi 0.2.3 ecmwf-api-client 1.5.4 ecmwfr 1.2.1
25. 3 TRAINING and REPRODUCIBLE workflows and tutorials NEED SAVE THE DATE: Reproducibility workshop @ECMWF | 14-16 Oct 2019
26. CLOUD is a PARADIGM-CHANGE... … and CHANGE takes time! 68% (Very) interested to migrate to cloud services Wagemann et al. (2019): in preparation
27. 1 “Problems cannot be solved by the same level of thinking that created them!” (Albert Einstein) MINDSET CHALLENGE
28. 2 Make REPRODUCIBILITY and SHARING to a part of your personal CODE OF CONDUCT Mindset challenge
29. 3 TRAIN others; SHARE your SKILLS and KNOWLEDGE! MINDSET CHALLENGE
30. Thus: BUILD BRIDGES Wish you an inspiring FOSS4G 2019
31. Some resources: • Climate Data Store | https://cds.climate.copernicus.eu/#!/home • ECMWF Public Datasets | https://apps.ecmwf.int/datasets/ • Global Flood Awareness System | http://www.globalfloods.eu/ • Fire danger forecasts | https://zenodo.org/communities/wildfire/ • ECMWF WMS Service endpoint | http://apps.ecmwf.int/wms/?token=public&request=GetCapabilities&version=1.3.1 • WEkEO Copernicus Cloud service | https://www.wekeo.eu/ • ECMWF is hiring! (https://www.ecmwf.int/en/about/jobs/jobs-ecmwf)