Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Open Meteorological and Climate Data - Building bridges between user communities

Open Meteorological and Climate Data - Building bridges between user communities

This presentation was given for a keynote at the FOSS4G 2019 conference in Bucharest, Romania on 29 August 2019.

The talk gives an overview of freely available environmental datasets from the European Centre for Medium-Range Weather Forecasts as part of the Copernicus programme. It covers aspects of open data and the importance of making them not just open, but also accessible and useable. The talk highlights the perspective of a data user trying to work with large volumes of meteorological and climate data.

Julia Wagemann

August 29, 2019
Tweet

More Decks by Julia Wagemann

Other Decks in Technology

Transcript

  1. Open
    METEOROLOGICAL
    and
    CLIMATE DATA
    Julia Wagemann
    Geospatial data consultant,
    Visiting Scientist @ECMWF,
    PhD candidate @MarbugUniversity
    @JuliaWagemann
    jwagemann.github.io
    Building BRIDGES between user communities!

    View full-size slide

  2. CLIMATE
    ERA5 reanalysis
    ● 1979 - almost Near-Real-Time
    ○ (soon from 1950 onwards)
    ● Temporal resolution: hourly
    ● Spatial resolution: ~ 30 km
    (ERA5-Land: 9 km)
    Seasonal forecasts
    ● monthly forecasts up to 6 months
    ahead
    ● Spatial resolution: 10 km

    View full-size slide

  3. CAMS reanalysis
    ● 2003 - 2016 (will be
    extended
    ● Spatial resolution: 80 km
    CAMS Forecasts
    ● 3-hourly forecasts up to
    5 days in advance
    ● Spatial resolution: 80 km
    Parameters: Ozone, CO2, SO2,
    NO2, ...
    AIR QUALITY

    View full-size slide

  4. Fire danger indices
    ● Fire-Weather Index
    ● Fine Fuel Moisture Code
    ● Duff Moisture Code
    ● Drought Code
    ● Build-Up Index
    ● Initial Spread Index
    ● Daily Severity Rating
    Time period: 1 Jan 1980 - 31 Dec 2018
    Temporal resolution: daily
    Spatial resolution: 80 km
    FIRE DANGER

    View full-size slide

  5. FLOOD
    River discharge
    ● GloFAS 30-day forecast
    ○ 1 Jan 2018 - realtime
    ○ Temporal resolution: daily
    ○ Spatial resolution: 10 km
    ● GloFAS 30-day reanalysis
    ○ 1981-2017
    ○ Temporal resolution: daily
    ○ Spatial resolution: 10 km

    View full-size slide

  6. Copernicus
    FULL, FREE and OPEN
    data policy

    View full-size slide

  7. *Aspects like
    FORMAT,
    STRUCTURE,
    MACHINE-READABILITY
    make the data
    MORE USABLE
    But these
    DO NOT
    make the data
    MORE OPEN!
    *European Data Portal (2019): elearning module #1 - What is ‘open’ data?

    View full-size slide

  8. Do aspects of
    non-interoperability,
    data complexity,
    community-specific formats, …
    make the data
    LESS OPEN?
    u

    View full-size slide

  9. CLIMATE
    OGC
    WCS
    Array
    database

    View full-size slide

  10. BIG DATA
    is
    NOT A NEW TERM
    1

    View full-size slide

  11. OPERATIONAL
    means ‘really’
    OPERATIONAL
    2
    24
    7
    /

    View full-size slide

  12. We talk about
    FIELDS and
    GRID POINTS
    not bands and pixels
    3

    View full-size slide

  13. STANDARDS and
    INTEROPERABILITY
    are
    NO NEW TERMS
    4

    View full-size slide

  14. 5
    We like
    CUSTOM-BUILT
    SOFTWARE
    developed in-house

    View full-size slide

  15. #1 Trend*
    *PwC for European Commission (2019): Copernicus market report
    (EO) expert user
    End user
    Intermediate user
    Non EO expert
    DIVERSIFICATION
    of users and their demands
    Decision-maker
    Policy maker
    Geospatial data user

    View full-size slide

  16. 1
    DATA
    COMPLEXITY
    CHALLENGE
    ENSEMBLES
    FORECASTS
    PRESSURE LEVELS

    View full-size slide

  17. User requirements
    survey
    Wagemann et al. (2019): in preparation
    1
    2
    3
    4

    View full-size slide

  18. How much of the
    ‘OPEN’
    DATA,
    that is produced, is
    USED?
    11 TBs forecast data
    per day
    10 TBs data from
    the Sentinels
    per day
    5 PBs of ERA5 data

    View full-size slide

  19. 1
    2
    Programmatic
    access
    Web
    Portal
    3
    FTP on
    request
    1 CDS
    Toolbox
    Download On-demand
    4
    Zenodo
    2 WMS

    View full-size slide

  20. MIRROR ARCHIVES as BRIDGES between
    different user communities
    Earth Observation community
    ERA5 data

    View full-size slide

  21. Example of NON-INTEROPERABILITY of systems:
    ERA5 in GEE
    Download Convert Upload Ingest
    SOON: 7 ERA5 parameters as part of GEE public data catalogue

    View full-size slide

  22. 1
    INTEROPERABILITY
    and
    STANDARD
    PROTOCOLS
    NEED
    WMS WCS WPS

    View full-size slide

  23. 2
    DRIVERS and
    PACKAGES for
    PYTHON and
    R
    NEED
    cfgrib 0.9.7.1
    Magics 1.0.7
    eccodes-python 0.9.2
    metview 1.2.0
    cdsapi 0.2.3
    ecmwf-api-client 1.5.4
    ecmwfr 1.2.1

    View full-size slide

  24. 3
    TRAINING and
    REPRODUCIBLE
    workflows and tutorials
    NEED
    SAVE THE DATE:
    Reproducibility workshop @ECMWF | 14-16 Oct 2019

    View full-size slide

  25. CLOUD is a PARADIGM-CHANGE...
    … and CHANGE takes time!
    68% (Very) interested
    to migrate to cloud
    services
    Wagemann et al. (2019): in preparation

    View full-size slide

  26. 1
    “Problems cannot be solved
    by the same level of thinking
    that created them!”
    (Albert Einstein)
    MINDSET CHALLENGE

    View full-size slide

  27. 2
    Make
    REPRODUCIBILITY and
    SHARING
    to a part of your personal
    CODE OF CONDUCT
    Mindset challenge

    View full-size slide

  28. 3
    TRAIN others;
    SHARE your
    SKILLS
    and
    KNOWLEDGE!
    MINDSET CHALLENGE

    View full-size slide

  29. Thus: BUILD BRIDGES
    Wish you an inspiring FOSS4G 2019

    View full-size slide

  30. Some resources:
    ● Climate Data Store | https://cds.climate.copernicus.eu/#!/home
    ● ECMWF Public Datasets | https://apps.ecmwf.int/datasets/
    ● Global Flood Awareness System | http://www.globalfloods.eu/
    ● Fire danger forecasts | https://zenodo.org/communities/wildfire/
    ● ECMWF WMS Service endpoint |
    http://apps.ecmwf.int/wms/?token=public&request=GetCapabilities&version=1.3.1
    ● WEkEO Copernicus Cloud service | https://www.wekeo.eu/
    ● ECMWF is hiring! (https://www.ecmwf.int/en/about/jobs/jobs-ecmwf)

    View full-size slide