Using Python in weather forecasting

Transcript

1. © ECMWF 14 November, 2018 Working with weather/climate data in Python Building a Python framework to interact with ECMWF data & services Stephan Siemen, Sylvie Lamy-Thepaut Development Section, ECMWF
2. Before we get started … About me • Head of Development Section • Veteran software developer Java, Fortran, C++, Python • Main interest is visualisations: IRIS Explorer, OpenGL/Inventor, mapping of weather forecasts • Twitter: @StephanSiemen • GitHub: StephanSiemen 2 EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS About the Development Section • > 20 developers • Skill sets: C++ / Python / web + DBs • Provide core software packages for ECMWF’s operations and wider community • Develop software to en/decode meteorological data formats (GRIB, BUFR, ODB, …), archive, process & visualise data • Develop time-critical data delivery & web services for our users
3. European Centre for Medium-range Weather Forecasts • Intergovernmental Organisation – 22 Member Sates + 12 Cooperation States – Headquarter in Reading, UK + Bologna, Italy – 350+ staff • 24/7 operational services – Operating weather forecast models – Data services delivering TBs of output data to user in real time world wide – HPC centre with two supercomputer clusters – Operate archive of observations and model data (> 250 PB) 3 EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS
4. Why we need a global model for medium-range (3-15 days)? 4 EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS
5. What does ECMWF provide? • Operational / time critical weather prediction – Medium-range – Extended-range (months) – Long-range (seasonal) • Ensembles of forecasts 5 EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS
6. How does it work? 6 EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS Archive MARS Data acquisition Product Generation Integrated Forecast System (IFS) Data assimilation Web Services Model output (in GRIB) Observations Users
7. Supercomputers at ECMWF ECMWF has a long history of using High Performance Computing in Numerical Weather Prediction • 1978 – 1996: Cray (Cray-1A, XMP, YMP, C90, T3D) • 1996 – 2002: Fujitsu (VPP700, VPP700E, VPP5000) • 2002 – 2014: IBM (Power4, Power5, Power6, Power7) • 2014 : Cray (Cray-XC30) 7 EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS
8. What comes in? 8 EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS Surface stations Radiosonde balloons Polar, infrared Polar, microwave Geostationary, IR Aircraft
9. Working with model output • Raw model output structured in layers of spherical harmonics representation & reduced Gaussian grids – Not pixels as in satellites! – Need interpolation to estimate values between grid points • Very few users work with the raw grid data directly – Express uncertainties of forecasts from Ensembles – Time series of forecast at one location 9 EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS
10. 10 EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS TODAY TOMORROW volume type type volume OBSERVATIONS MODELS MODELS OBSERVATIONS 40 million = 4 x 107 98% from 60 different satellite instruments 100-200 million = 1-2 x 108 98% from 80 different satellite instruments 10 million grid points 100 levels 10 prognostic variables = 1 x 1010 physical parameters of atmosphere, waves, ocean physical and chemical parameters of atmosphere, waves, ocean, ice, vegetation 500 million grid points 200 levels 100 prognostic variables = 1 x 1013 And the data is constantly growing … FACTOR 5 FACTOR 1000
11. EU Copernicus programme • Large programme to make earth observations and monitoring data (freely) available • Data is validated and from verified sources • Sponsored by the European Commission • ECMWF is operating two services on behalf of the EC • Copernicus Atmospheric Monitoring Service (CAMS) – Air pollution, chemicals, sand & dust – https://atmosphere.copernicus.eu/data • Copernicus Climate Change Services (C3S) – Reanalysis data, seasonal forecasts, climate scenarios 11 EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS Total column of sulphur dioxide (provided by the Copernicus Atmosphere Monitoring Service)
12. Copernicus Climate Data Store (CDS) • New portal to find / download and work with Copernicus climate change data • Many data sets too large for users to work locally à therefore it offers server side processing • High-level descriptive Python interface – Allow non-domain users to build apps • Try it out yourself: https://cds.climate.copernicus.eu 12 EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS CDS toolbox
13. CDS toolbox in action 13 EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS
14. Building systems with Python 14 EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS
15. Using Python to give data access from archives • You need a token to use the service! • ECMWF data: https://pypi.org/project/ecmwf-api-client/ • Copernicus data: https://pypi.org/project/cdsapi/ 15 EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS
16. Describing workflows for operational suites - ecflow 16 EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS
17. What about the number crunching? 17 EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS • Numerical Weather Prediction (NWP) has always pushed (HPC) computing to its limits • Traditionally Fortran and increasingly C/C++ has played a big part and still is heavily used for time critical parts of the models • Python is mostly used in post-processing of model data • Key challenge is to read data easily and fast • Abstracting data in a common data model – data cube – Allows non-expert users to work with data – xarray is fundamental building block of this approach • xarray structure is based on the Common Data Model from netCDF
18. Active engagement with community We had now two workshops with wider Python community 18 EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS • There are already many good efforts and solutions out there à Many good “wheels” which do not need to be reinvented à We want to allow easy interactions between frameworks • Confirmation of our direction for developments à Using common Python packages for meteorological data à Handle fields through xarray; pandas for tables/time series à Build new Python interfaces the Python way à Not how legacy Fortran/C interfaces were done
19. Looking at data processing at scale 19 EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS • We currently experimenting how we can make better use of pandas, xarray & dask to scale post-processing with Python • We are now looking at how we can use Python itself for larger datasets and provide tutorials & cookbooks for users how to process our data • We had a very good session on this topic at the last workshop – E.g the work of the Pangeo initiative here is very relevant • These and many other interesting presentations from this year’s workshop you can find at https://www.ecmwf.int/en/learning/workshops/2018-workshop-developing-python-frameworks-earth-system-sciences
20. Benefitting the wider community cfgrib – linking xarray and ecCodes • Essential building block to bring GRIB model output data in the PyData stack • To embrace xarray for all our field data, we needed to know that we could handle all our GRIB data – Therefore it was important for us to have a solution based on our own GRIB decoder – Developed with our partners at • Open to the whole community – First user: CDS toolbox 20 EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS
21. xarray + cfgrib in action 21 EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS
22. Magics - plotting weather maps • High-level data aware plot package • Automatic styling & fast render • https://confluence.ecmwf.int/magics 22 EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS
23. 23 EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS The Metview Python framework • A high-level Python 3 interface for processing and visualising ECMWF data à makes use of all other packages • Aim is to allow users of Metview to use easily the power of Python but still have all functionality of Metview; including visualisation
24. Generation of Python code 24 EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS
25. Making software easily available within existing frameworks • Source code & examples on GitHub • Packages need be on PyPi and Conda • ECMWF Python software on DockerHub • CDS Toolbox 25 EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS
26. How can I get try our software right now? 26 EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS • Docker image on DockerHub – https://hub.docker.com/r/ecmwf/jupyter-notebook/ • Available on github and PyPi – e.g. https://github.com/ecmwf/metview-python – e.g. pip install metview • Check out our Jupyter notebook examples at https://github.com/ecmwf/notebook-examples
27. We are looking for help! • We have regular job openings – https://www.ecmwf.int/en/about/jobs/jobs-ecmwf • We also tender/outsource many of our developments – https://www.ecmwf.int/en/about/suppliers • We look for contractors – E.g. porting IDL programs to Python • Feel free to contribute/improve our Open Source software – Will benefit whole meteorological/climate community! – https://github.com/ecmwf 27 EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS
28. • Stipend of £5000 • Challenges will be published on GitHub on 21st Jan: https://github.com/esowc • Follow #ESoWC2019 for updates