Open Meteorological and Climate Data - Building bridges between user communities

Open Meteorological and Climate Data - Building bridges between user communities

This presentation was given for a keynote at the FOSS4G 2019 conference in Bucharest, Romania on 29 August 2019.

The talk gives an overview of freely available environmental datasets from the European Centre for Medium-Range Weather Forecasts as part of the Copernicus programme. It covers aspects of open data and the importance of making them not just open, but also accessible and useable. The talk highlights the perspective of a data user trying to work with large volumes of meteorological and climate data.

157d70ab1729f9665879a445fb6b7c87?s=128

Julia Wagemann

August 29, 2019
Tweet

Transcript

  1. Open METEOROLOGICAL and CLIMATE DATA Julia Wagemann Geospatial data consultant,

    Visiting Scientist @ECMWF, PhD candidate @MarbugUniversity @JuliaWagemann jwagemann.github.io Building BRIDGES between user communities!
  2. None
  3. CLIMATE ERA5 reanalysis • 1979 - almost Near-Real-Time ◦ (soon

    from 1950 onwards) • Temporal resolution: hourly • Spatial resolution: ~ 30 km (ERA5-Land: 9 km) Seasonal forecasts • monthly forecasts up to 6 months ahead • Spatial resolution: 10 km
  4. CAMS reanalysis • 2003 - 2016 (will be extended •

    Spatial resolution: 80 km CAMS Forecasts • 3-hourly forecasts up to 5 days in advance • Spatial resolution: 80 km Parameters: Ozone, CO2, SO2, NO2, ... AIR QUALITY
  5. Fire danger indices • Fire-Weather Index • Fine Fuel Moisture

    Code • Duff Moisture Code • Drought Code • Build-Up Index • Initial Spread Index • Daily Severity Rating Time period: 1 Jan 1980 - 31 Dec 2018 Temporal resolution: daily Spatial resolution: 80 km FIRE DANGER
  6. FLOOD River discharge • GloFAS 30-day forecast ◦ 1 Jan

    2018 - realtime ◦ Temporal resolution: daily ◦ Spatial resolution: 10 km • GloFAS 30-day reanalysis ◦ 1981-2017 ◦ Temporal resolution: daily ◦ Spatial resolution: 10 km
  7. Copernicus FULL, FREE and OPEN data policy

  8. *Aspects like FORMAT, STRUCTURE, MACHINE-READABILITY make the data MORE USABLE

    But these DO NOT make the data MORE OPEN! *European Data Portal (2019): elearning module #1 - What is ‘open’ data?
  9. Do aspects of non-interoperability, data complexity, community-specific formats, … make

    the data LESS OPEN? u
  10. CLIMATE OGC WCS Array database

  11. BIG DATA is NOT A NEW TERM 1

  12. OPERATIONAL means ‘really’ OPERATIONAL 2 24 7 /

  13. We talk about FIELDS and GRID POINTS not bands and

    pixels 3
  14. STANDARDS and INTEROPERABILITY are NO NEW TERMS 4

  15. 5 We like CUSTOM-BUILT SOFTWARE developed in-house

  16. #1 Trend* *PwC for European Commission (2019): Copernicus market report

    (EO) expert user End user Intermediate user Non EO expert DIVERSIFICATION of users and their demands Decision-maker Policy maker Geospatial data user
  17. 1 DATA COMPLEXITY CHALLENGE ENSEMBLES FORECASTS PRESSURE LEVELS

  18. User requirements survey Wagemann et al. (2019): in preparation 1

    2 3 4
  19. How much of the ‘OPEN’ DATA, that is produced, is

    USED? 11 TBs forecast data per day 10 TBs data from the Sentinels per day 5 PBs of ERA5 data
  20. 1 2 Programmatic access Web Portal 3 FTP on request

    1 CDS Toolbox Download On-demand 4 Zenodo 2 WMS
  21. MIRROR ARCHIVES as BRIDGES between different user communities Earth Observation

    community ERA5 data
  22. Example of NON-INTEROPERABILITY of systems: ERA5 in GEE Download Convert

    Upload Ingest SOON: 7 ERA5 parameters as part of GEE public data catalogue
  23. 1 INTEROPERABILITY and STANDARD PROTOCOLS NEED WMS WCS WPS

  24. 2 DRIVERS and PACKAGES for PYTHON and R NEED cfgrib

    0.9.7.1 Magics 1.0.7 eccodes-python 0.9.2 metview 1.2.0 cdsapi 0.2.3 ecmwf-api-client 1.5.4 ecmwfr 1.2.1
  25. 3 TRAINING and REPRODUCIBLE workflows and tutorials NEED SAVE THE

    DATE: Reproducibility workshop @ECMWF | 14-16 Oct 2019
  26. CLOUD is a PARADIGM-CHANGE... … and CHANGE takes time! 68%

    (Very) interested to migrate to cloud services Wagemann et al. (2019): in preparation
  27. 1 “Problems cannot be solved by the same level of

    thinking that created them!” (Albert Einstein) MINDSET CHALLENGE
  28. 2 Make REPRODUCIBILITY and SHARING to a part of your

    personal CODE OF CONDUCT Mindset challenge
  29. 3 TRAIN others; SHARE your SKILLS and KNOWLEDGE! MINDSET CHALLENGE

  30. Thus: BUILD BRIDGES Wish you an inspiring FOSS4G 2019

  31. Some resources: • Climate Data Store | https://cds.climate.copernicus.eu/#!/home • ECMWF

    Public Datasets | https://apps.ecmwf.int/datasets/ • Global Flood Awareness System | http://www.globalfloods.eu/ • Fire danger forecasts | https://zenodo.org/communities/wildfire/ • ECMWF WMS Service endpoint | http://apps.ecmwf.int/wms/?token=public&request=GetCapabilities&version=1.3.1 • WEkEO Copernicus Cloud service | https://www.wekeo.eu/ • ECMWF is hiring! (https://www.ecmwf.int/en/about/jobs/jobs-ecmwf)