Open Meteorological and Climate Data - Building bridges between user communities

This presentation was given for a keynote at the FOSS4G 2019 conference in Bucharest, Romania on 29 August 2019.

The talk gives an overview of freely available environmental datasets from the European Centre for Medium-Range Weather Forecasts as part of the Copernicus programme. It covers aspects of open data and the importance of making them not just open, but also accessible and useable. The talk highlights the perspective of a data user trying to work with large volumes of meteorological and climate data.

Transcript

1. Open
   METEOROLOGICAL
   and
   CLIMATE DATA
   Julia Wagemann
   Geospatial data consultant,
   Visiting Scientist @ECMWF,
   PhD candidate @MarbugUniversity
   @JuliaWagemann
   jwagemann.github.io
   Building BRIDGES between user communities!
   

   View Slide

2. View Slide

3. CLIMATE
   ERA5 reanalysis
   ● 1979 - almost Near-Real-Time
   ○ (soon from 1950 onwards)
   ● Temporal resolution: hourly
   ● Spatial resolution: ~ 30 km
   (ERA5-Land: 9 km)
   Seasonal forecasts
   ● monthly forecasts up to 6 months
   ahead
   ● Spatial resolution: 10 km
   

   View Slide

4. CAMS reanalysis
   ● 2003 - 2016 (will be
   extended
   ● Spatial resolution: 80 km
   CAMS Forecasts
   ● 3-hourly forecasts up to
   5 days in advance
   ● Spatial resolution: 80 km
   Parameters: Ozone, CO2, SO2,
   NO2, ...
   AIR QUALITY
   

   View Slide

5. Fire danger indices
   ● Fire-Weather Index
   ● Fine Fuel Moisture Code
   ● Duff Moisture Code
   ● Drought Code
   ● Build-Up Index
   ● Initial Spread Index
   ● Daily Severity Rating
   Time period: 1 Jan 1980 - 31 Dec 2018
   Temporal resolution: daily
   Spatial resolution: 80 km
   FIRE DANGER
   

   View Slide

6. FLOOD
   River discharge
   ● GloFAS 30-day forecast
   ○ 1 Jan 2018 - realtime
   ○ Temporal resolution: daily
   ○ Spatial resolution: 10 km
   ● GloFAS 30-day reanalysis
   ○ 1981-2017
   ○ Temporal resolution: daily
   ○ Spatial resolution: 10 km
   

   View Slide

7. Copernicus
   FULL, FREE and OPEN
   data policy
   

   View Slide

8. *Aspects like
   FORMAT,
   STRUCTURE,
   MACHINE-READABILITY
   make the data
   MORE USABLE
   But these
   DO NOT
   make the data
   MORE OPEN!
   *European Data Portal (2019): elearning module #1 - What is ‘open’ data?
   

   View Slide

9. Do aspects of
   non-interoperability,
   data complexity,
   community-specific formats, …
   make the data
   LESS OPEN?
   u
   

   View Slide

10. CLIMATE
    OGC
    WCS
    Array
    database
    

    View Slide

11. BIG DATA
    is
    NOT A NEW TERM
    1
    

    View Slide

12. OPERATIONAL
    means ‘really’
    OPERATIONAL
    2
    24
    7
    /
    

    View Slide

13. We talk about
    FIELDS and
    GRID POINTS
    not bands and pixels
    3
    

    View Slide

14. STANDARDS and
    INTEROPERABILITY
    are
    NO NEW TERMS
    4
    

    View Slide

15. 5
    We like
    CUSTOM-BUILT
    SOFTWARE
    developed in-house
    

    View Slide

16. #1 Trend*
    *PwC for European Commission (2019): Copernicus market report
    (EO) expert user
    End user
    Intermediate user
    Non EO expert
    DIVERSIFICATION
    of users and their demands
    Decision-maker
    Policy maker
    Geospatial data user
    

    View Slide

17. 1
    DATA
    COMPLEXITY
    CHALLENGE
    ENSEMBLES
    FORECASTS
    PRESSURE LEVELS
    

    View Slide

18. User requirements
    survey
    Wagemann et al. (2019): in preparation
    1
    2
    3
    4
    

    View Slide

19. How much of the
    ‘OPEN’
    DATA,
    that is produced, is
    USED?
    11 TBs forecast data
    per day
    10 TBs data from
    the Sentinels
    per day
    5 PBs of ERA5 data
    

    View Slide

20. 1
    2
    Programmatic
    access
    Web
    Portal
    3
    FTP on
    request
    1 CDS
    Toolbox
    Download On-demand
    4
    Zenodo
    2 WMS
    

    View Slide

21. MIRROR ARCHIVES as BRIDGES between
    different user communities
    Earth Observation community
    ERA5 data
    

    View Slide

22. Example of NON-INTEROPERABILITY of systems:
    ERA5 in GEE
    Download Convert Upload Ingest
    SOON: 7 ERA5 parameters as part of GEE public data catalogue
    

    View Slide

23. 1
    INTEROPERABILITY
    and
    STANDARD
    PROTOCOLS
    NEED
    WMS WCS WPS
    

    View Slide

24. 2
    DRIVERS and
    PACKAGES for
    PYTHON and
    R
    NEED
    cfgrib 0.9.7.1
    Magics 1.0.7
    eccodes-python 0.9.2
    metview 1.2.0
    cdsapi 0.2.3
    ecmwf-api-client 1.5.4
    ecmwfr 1.2.1
    

    View Slide

25. 3
    TRAINING and
    REPRODUCIBLE
    workflows and tutorials
    NEED
    SAVE THE DATE:
    Reproducibility workshop @ECMWF | 14-16 Oct 2019
    

    View Slide

26. CLOUD is a PARADIGM-CHANGE...
    … and CHANGE takes time!
    68% (Very) interested
    to migrate to cloud
    services
    Wagemann et al. (2019): in preparation
    

    View Slide

27. 1
    “Problems cannot be solved
    by the same level of thinking
    that created them!”
    (Albert Einstein)
    MINDSET CHALLENGE
    

    View Slide

28. 2
    Make
    REPRODUCIBILITY and
    SHARING
    to a part of your personal
    CODE OF CONDUCT
    Mindset challenge
    

    View Slide

29. 3
    TRAIN others;
    SHARE your
    SKILLS
    and
    KNOWLEDGE!
    MINDSET CHALLENGE
    

    View Slide

30. Thus: BUILD BRIDGES
    Wish you an inspiring FOSS4G 2019
    

    View Slide

31. Some resources:
    ● Climate Data Store | https://cds.climate.copernicus.eu/#!/home
    ● ECMWF Public Datasets | https://apps.ecmwf.int/datasets/
    ● Global Flood Awareness System | http://www.globalfloods.eu/
    ● Fire danger forecasts | https://zenodo.org/communities/wildfire/
    ● ECMWF WMS Service endpoint |
    http://apps.ecmwf.int/wms/?token=public&request=GetCapabilities&version=1.3.1
    ● WEkEO Copernicus Cloud service | https://www.wekeo.eu/
    ● ECMWF is hiring! (https://www.ecmwf.int/en/about/jobs/jobs-ecmwf)
    

    View Slide