Speeding up Programs with OpenACC in GCC

In this deck from FOSDEM 2020, John Hanley from ECMWF presents: Speeding up Programs with OpenACC in GCC.

"Weather forecasts produced by ECMWF and environment services by the Copernicus programme act as a vital input for many downstream simulations and applications. A variety of products, such as ECMWF reanalyses and archived forecasts, are additionally available to users via the MARS archive and the Copernicus data portal. Transferring, storing and locally modifying large volumes of such data prior to integration currently presents a significant challenge to users. The key aim for ECMWF within the H2020 HiDALGO project is to migrate some of these tasks to the cloud, thereby facilitating fast and seamless application integration by enabling precise and efficient data delivery to the end-user. The required cloud infrastructure development will also feed into ECMWF's contribution to the European Weather Cloud pilot which is a collaborative cloud development project between ECMWF and EUMETSAT.

ECMWF and its HiDALGO partners aim to implement a set of services that enable the simulation of complex global challenges which require massive high performance computing resources alongside state-of-the-art data analytics and visualization. ECMWF's role in the project will be to enable seamless integration of two pilot applications with its meteorological data and services delivered via ECMWF's Cloud and orchestrated by bespoke HiDALGO workflows. The demonstrated workflows show the increased value of weather forecasts, but also derived forecasts for air quality as provided by the Copernicus Atmospheric Monitoring Service (CAMS).

The HiDALGO use-case workflows are comprised of four main components: pre-processing, numerical simulation, post-processing and visualization. The core simulations are ideally suited to running in a dedicated HPC environment, due to their large computational demands, coupled with the heavy communication overhead between parallel processes. However, the pre-/post-processing and visualisation tasks generally do not demand more than a few cores to compute and do not require message passing between instances, hence they are good candidates to run in a cloud environment. Enabling, managing and orchestrating the integration of both HPC and cloud environments to improve overall performance is the key goal of HiDALGO.

This talk will give a general overview of HiDALGO project and its main aims and objectives. It will present the two test pilot applications which will be used for integration, and an overview of the general workflows and services within HiDALGO. In particular, it will focus on how ECMWF's cloud data and services will couple with the test pilot applications to improve overall workflow performance and enable access to new data for the pilot users."

Watch the video: https://wp.me/p3RLHQ-lwd

Learn more: https://hidalgo-project.eu/
and
https://fosdem.org/2020/schedule/

Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter

Transcript

1. HiDALGO – EU founded project #824115 Building Cloud-Based Data Services to Enable Earth Science Workflows across HPC Centres John Hanley, Milana Vuckovic, James Hawkes, Tiago Quintino, Stephan Siemen, Florian Pappenberger John.Hanley@ecmwf.int
2. HiDALGO – EU founded project #824115 Introduction to ECMWF The Data Challenge HiDALGO & ECMWF Overview
3. 3 02.02.2020 HiDALGO 3 European Centre for Medium-Range Weather Forecasts • Operates two Copernicus Services • Climate Change Service (C3S) • Atmosphere Monitoring Service (CAMS) • Established in 1975. • Intergovernmental Organisation • 22 Member States | 12 Cooperation States • 350+ staff • 24/7 operational service • Operational NWP centre • Supporting NWS (coupled models) and businesses • Research institution • Closely connected with researchers worldwide • Supports Copernicus Emergency Management Service (CEMS)
4. 4 02.02.2020 HiDALGO 4 What do we do? Short-range weather forecast Very high resolution Regional models 1-2 hour production schedule Medium-range weather forecast High resolution Global models 6-12 hour production schedule Long-range weather forecast Predicts statistics of weather for coming month or season 1-8 times a month production schedule Climate prediction CO2 doubling and other scenarios
5. 02.02.2020 HiDALGO 5 What do we do? Operations – Time Critical – HRES 0-10 day, 00Z+12Z, 9km @ 137 levels – ENS 0-15 day, 00Z+12Z, 18km @ 91 levels – BC 06Z and 18Z, 0-5 days hourly – 100 TiB, 85 Million products – Real-time Dissemination, 200 destinations world-wide Research – Non Time Critical – 100s Daily active experiments to improve our models – Reforecasts, Climate reanalysis, etc Meteorological Archive – > 300 PiB of data @ 5000 daily active users – 250 TiB added per day
6. 6 02.02.2020 HiDALGO 6 Cloud • [SaaS] Copernicus Data Storage (CDS) – Operational • [PaaS] European Weather Cloud – Pilot currently being setup • [XaaS] WEkEO www.wekeo.eu Archive Largest Meteo archive 4x Oracle (Sun) SL8500 tape libraries ~ 140 Tape drives + 100 TiB / day operational + 150 TiB / day other HPC 2x Cray XC40 HPC 2x 129,960 cores Xeon EP E5-2695 Broadwell 2x 10 PiB Lustre PFS storage Top500 42nd/43rd ECMWF’s Facilities
7. HiDALGO – EU founded project #824115 Introduction to ECMWF The Data Challenge HiDALGO & ECMWF
8. 8 ECMWF Data Growth – History and Projections Historical Growth of Generated Products Model Output Projected Growth 02.02.2020 HiDALGO 8 • Data archival and retrieval system for ECMWF data – > 300 PB primary data • Largest meteorological archive in the world – Direct access from Member States – Available to research community worldwide
9. Types of data growth 02.02.2020 HiDALGO 9 Ensembles ➔Reliability ➔Accuracy Traditional weather science domain ➔Range Traditional climate science domain Today: we need high-resolution, ‘Earth system’ model ensembles to perform at all scales!
10. 02.02.2020 HiDALGO 10 The data challenge • No user can handle all our data in real-time – Much of ECMWF (Ensemble) forecast stays unused! – ECMWF always looks for new ways to give user access to more of its forecasts – Not made easier by domain specific formats & conventions • Dissemination system – Sophisticated push system to disseminate 100TBs in real time across the world • Web services – Develop & explore (GIS/OGC) web services to allow users to request data on-demand The Key Challenge: How do we improve user access to such volumes of data?
11. 02.02.2020 HiDALGO 11 How can you access data today? • Try it out yourself: https://pypi.org/project/ecmwf-api-client/ https://apps.ecmwf.int/datasets/
12. 02.02.2020 HiDALGO 12 How can you access data today? Copernicus Climate Data Store (CDS) • New portal to find / download and work with Copernicus climate change data CDS toolbox • Many data sets too large for users to work locally → therefore it offers server side processing • High-level descriptive Python interface – Allow non-domain users to build apps • Try it out yourself: https://cds.climate.copernicus.eu
13. PFS Cloud HDD Tape MARS FDB Consumer Archive • Bring users to the data • Use data while it is hot • Access using scientifically meaningful metadata Producer 02.02.2020 HiDALGO 13 ECMWF Novel Data Flows
14. HiDALGO – EU founded project #824115 Introduction to ECMWF The Data Challenge HiDALGO & ECMWF
15. HiDALGO 15 HiDALGO: HPC and Big Data Technologies for Global Systems – European project funded by the Horizon 2020 Framework Programme of the European Union carried out by 13 institutions from seven countries. The Vision: To advance technology to master global challenges The Mission: To develop novel methods, algorithms and software for HPC & HPDA to accurately model and simulate the related complex processes. To also enable coupling of pilots to external data sources (e.g. ECMWF). Pilot Test Cases: 1. Migration pilot (Derek Groen, Brunel University, UK). 2. Air pollution pilot (Zoltán Horváth, SZE, Hungary) 3. Social networks pilot (Robert Elsässer, PLUS, Austria) 02.02.2020
16. 02.02.2020 HiDALGO 16 ECMWF’s role Enable coupling as a means to build a workflow With a "closed" HPC system, ECMWF brings in valuable experience on how these systems can be integrated in larger workflows --> this can be a model for many similar HPC systems around Europe!
17. 02.02.2020 HiDALGO 17 HiDALGO HPC & Cloud Facilities • Cray XC40 “Hazelhen” • 7.4 PFLOPS • 185,088 cores • Huawei CH121 “Eagle” • 1.4 PFLOPS • 32,984 cores Cloud
18. HiDALGO 18 Weather and Climate Data Coupling Two step approach to coupling Step 2: Dynamic coupling (2nd year – 2020 onwards) - coupling with forecast data via a REST API Cloud • Bring users to the data • Use data while it is hot • Access using scientifically meaningful metadata Consumer Vision: To enable users to build custom workflows utilizing ECMWF's weather forecast and climate data Step 1: Static coupling (1st year of project - 2019) - coupling with static reanalysis (climate) data for the purposes of pilot model calibration Completed! Climate Data Store (CDS) 02.02.2020
19. 19 © HiDALGO The main requirements: 1. Bring users to the data and avoid moving the data out of the data centre. 2. Provide users with computing resources collocated directly with data. 3. Align with data-centric approach of “move the compute, not the data”. How to enable this: 1. Mechanism to pull/push data from ECMWF. 2. Mechanism to run custom post-processing at ECMWF. 3. Mechanism to explore what data and processing options ECMWF offers. Providing ECMWF data to the pilot applications push pull 02.02.2020
20. 20 © HiDALGO Cloud Data-as-a-Service: Polytope Request ID (202 ACCEPTED) polytope retrieve <request> (POST) • Under development at ECMWF • Deployed internally at ECMWF • Accessible externally • Beta-tested via European Weather Cloud • Exposes a RESTful API • A CLI and python API aid the users interacting with the Polytope API • It interfaces MARS directly • Will implement hyper-cube data access Service designed for efficient provisioning of meteorological data to Cloud and HPC applications { 'stream' : 'oper', 'type' : 'an', 'class' : 'ei', 'dataset' : 'interim', 'levtype' : 'sfc', 'param' : '165.128’, ... } Polytope Client Polytope Server Step 1: submit request 02.02.2020
21. 21 © HiDALGO Cloud Data-as-a-Service: Polytope Service designed for efficient provisioning of meteorological data to Cloud and HPC applications Polytope Client Polytope Server polytope list requests (GET) Request IDs (200 OK) Optional step: list requests 02.02.2020 • Under development at ECMWF • Deployed internally at ECMWF • Accessible externally • Beta-tested via European Weather Cloud • Exposes a RESTful API • A CLI and python API aid the users interacting with the Polytope API • It interfaces MARS directly • Will implement hyper-cube data access
22. 22 © HiDALGO Cloud Data-as-a-Service: Polytope Service designed for efficient provisioning of meteorological data to Cloud and HPC applications Polytope Client Polytope Server Request ID (GET) Data (200 OK) Step 2: poll for data 02.02.2020 • Under development at ECMWF • Deployed internally at ECMWF • Accessible externally • Beta-tested via European Weather Cloud • Exposes a RESTful API • A CLI and python API aid the users interacting with the Polytope API • It interfaces MARS directly • Will implement hyper-cube data access
23. 23 © HiDALGO Cloud Data-as-a-Service: Polytope Service designed for efficient provisioning of meteorological data to Cloud and HPC applications Polytope Client Polytope Server polytope revoke <id> (DELETE) (200 OK) Step 3: delete completed request 02.02.2020 • Under development at ECMWF • Deployed internally at ECMWF • Accessible externally • Beta-tested via European Weather Cloud • Exposes a RESTful API • A CLI and python API aid the users interacting with the Polytope API • It interfaces MARS directly • Will implement hyper-cube data access
24. 24 © HiDALGO Cloud Data-as-a-Service: Polytope FRONTEND request (POST) api/v1/requests api/v1/requests/ <request_id> Data (200 OK) ID (GET) . . . request store FDB MARS OTHER SOURCE data staging worker Worker Pool Data Sources The system has been developed as a set of independent services for scalability (elastic architecture, muti frontend, workers, etc.), ease of deployment (Kubernetes support), with a shallow software stack. Queue ||||||||||||||||||| polytope.ecmwf.int broker ID (202 ACCEPTED) 02.02.2020
25. 25 THANK YOU ! QUESTIONS ? HiDALGO – EU founded project #824115 Acknowledgements This work has been supported by the HiDALGO project and has been partly funded by the European Commission’s ICT activity of the H2020 Programme under grant agreement number: 824115. This paper expresses the opinions of the authors and not necessarily those of the European Commission. The European Commission is not liable for any use that may be made of the information contained in this paper. 02.02.2020