EGI user support & interaction with the VRCs

Transcript

1. www.egi.eu EGI-InSPIRE RI-261323 EGI-InSPIRE www.egi.eu EGI-InSPIRE RI-261323 EGI user support & interaction with the VRCs Nuno Ferreira User Community Support Team Officer @ EGI.eu WeNMR Devs Workshop, Utrecht, NL 20110712/13 1
2. www.egi.eu EGI-InSPIRE RI-261323 • The grid: a primer • Managing a sustainable infrastructure • Building sustainable user services • Summary Outline WeNMR Devs Workshop, Utrecht, NL 20110712/13 2
3. www.egi.eu EGI-InSPIRE RI-261323 What is the grid? “Coordinated resource sharing and problem solving in dynamic, multi-institutional virtual organizations”. Foster, I. et al., Int. J. Superc. Appli. (2000)15:3 WeNMR Devs Workshop, Utrecht, NL 20110712/13 3 Source: http://www.gridcafe.org
4. www.egi.eu EGI-InSPIRE RI-261323 Infrastructure (Wikipedia) Infrastructure is the basic physical and organisational structures needed for the operation of a society or enterprise, or the services and facilities necessary for an economy to function European Grid Infrastructure (EGI) provides a service infrastructure that exposes and helps coordinate a resource infrastructure WeNMR Devs Workshop, Utrecht, NL 20110712/13 4 The Enterprise is the European Research Area
5. www.egi.eu EGI-InSPIRE RI-261323 • The grid: a primer • Managing a sustainable infrastructure • Building sustainable user services • Summary Outline WeNMR Devs Workshop, Utrecht, NL 20110712/13 5
6. www.egi.eu EGI-InSPIRE RI-261323 EGI Resource Infrastructure WeNMR Devs Workshop, Utrecht, NL 20110712/13 6 Resource Infrastructure Resource Centres Resource Centres Resource Infrastructure Resource Centres Resource Centres Resource Infrastructure Resource Centres Resource Centres Network Integrated Resource Infrastructure Provider National Grid Initiative Peer Resource Infrastructure Provider Integrated: infrastructure operated by a non-EGI-InSPIRE partner but relying on EGI operational services (MoU) Peer: infrastructure accessible to EGI users, but relying on own operational services MoUs EGI.eu
7. www.egi.eu EGI-InSPIRE RI-261323 EGI Collaboration Context & NA2 - EGI-InSPIRE Review 2011 7 NGI: National Grid Initiative EIRO: European Intergovernmental Research Organisation NGI NGI EIRO NGI EIRO Virtual Research Community NGI NGI Virtual Research Community Virtual Research Community EGI.eu
8. www.egi.eu EGI-InSPIRE RI-261323 Status and yearly increase WeNMR Devs Workshop, Utrecht, NL 20110712/13 8 Resource Centres • 338 (+6.8%) : Europe, Asia Pacific, North and South America • 96 supporting MPI (+31.5%) Countries • 51 (57 with integrated RPs) (+18.75%) Capacity • 240,000 CPU cores • 339,000 with integrated and peer infrastructures (24.9%) • 1.89 Million HEP-SPEC 06* • 102 PB disk, 89 PB tape * HEP-SPEC 06: Computing benchmark based on SPECCPU2006, 10 HEP-SPEC = 4 kSI2k
9. www.egi.eu EGI-InSPIRE RI-261323 EGI.eu • Coordination for European Grid resources – Established February 8th 2010 – Central policy & services needed to run a grid – Sustainable small coordinating organisation • Based in Amsterdam – Coordinating core (~20 people) in Amsterdam – Technical services from partners (~20 people) WeNMR Devs Workshop, Utrecht, NL 20110712/13 9 EGI and EGI.eu supported by EGI-InSPIRE project
10. www.egi.eu EGI-InSPIRE RI-261323 EGI-InSPIRE project Integrated Sustainable Pan-European Infrastructure for Researchers in Europe A 4 year project with €25M EC contribution – Project cost €72M – Total Effort ~€330M – Effort: 9261PMs Project Partners (50) EGI.eu, 38 NGIs, 2 EIROs Asia Pacific (9 partners) Un-Funded WeNMR Devs Workshop, Utrecht, NL 20110712/13 10 Funded
11. www.egi.eu EGI-InSPIRE RI-261323 EGI-InSPIRE objectives • A sustainable production infrastructure – Resource providers in Europe and worldwide – With new technologies as they mature • Support structured international research – Sustain current domain specific services – Attract new user communities (e.g. ESFRI) WeNMR Devs Workshop, Utrecht, NL 20110712/13 11
12. www.egi.eu EGI-InSPIRE RI-261323 • The grid: a primer • Managing a sustainable infrastructure • Building sustainable user services • Summary Outline WeNMR Devs Workshop, Utrecht, NL 20110712/13 12
13. www.egi.eu EGI-InSPIRE RI-261323 A virtuous service cycle WeNMR Devs Workshop, Utrecht, NL 20110712/13 Used by Researchers Gathering New Requirements New Technology Assessed Deployed Infrastructure Services 13 EGI.eu National Resource Providers Current User Communities New User Communities Technology Providers
14. www.egi.eu EGI-InSPIRE RI-261323 UCB and the VRC model WeNMR Devs Workshop, Utrecht, NL 20110712/13 14 Benefits: 1. Defined communication channels: named contact points between communities & EGI 2. Application integration 3. Policy and procedures: define interaction between EGI and users 4. Requirements gathering: solutions and influence 5. Dissemination: sharing of news, breakthroughs, best practice and innovation User Community Board (UCB) with representation from: Life Sciences, WLCG, WeNMR, Astronomy & Astrophysics, Earth Sciences, Computational Chemistry, Arts & Humanities, Hydrometeorology VO: a group of researchers working collaboratively EGI Helpdesk Other Helpdesk E G I. e u N G I Training Events Trainers Technical Services VRC support units Other support units UCB User Community VRC: a grouping of research collaborations working in a common domain
15. www.egi.eu EGI-InSPIRE RI-261323 NGI User Support Teams WeNMR Devs Workshop, Utrecht, NL 20110712/13 15 NGI User Support Teams Consultancy Training Provide access to grid and software services Porting apps. Dev. and operate software services Collecting Feedback Document Processes & apps. Helpdesk NGI User Support Teams provide: • Consultancy & training • Access to grid services • Application porting • Developing and operating software services • Collecting feedback • Documentation & helpdesk NGIs receive: • Web & wiki resources • Requirements process • Applications and services • VRC coordination support Support teams are available across 50 partners – they are all different… as are their support skills.
16. www.egi.eu EGI-InSPIRE RI-261323 NGI User Support Teams WeNMR Devs Workshop, Utrecht, NL 20110712/13 16 NGI User Support Teams Consultancy Training Provide access to grid and software services Porting apps. Dev. and operate software services Collecting Feedback Document Processes & apps. Helpdesk NGI User Support Teams provide: • Consultancy & training • Access to grid services • Application porting • Developing and operating software services • Collecting feedback • Documentation & helpdesk NGIs receive: • Web & wiki resources • Requirements process • Applications and services • VRC coordination support Support teams are available across 50 partners – they are all different… as are their support skills.
17. www.egi.eu EGI-InSPIRE RI-261323 Technical support services WeNMR Devs Workshop, Utrecht, NL 20110712/13 17 Services provided by a few NGIs for the whole community • To support the Virtual Research Communities • To facilitate the work of NGI support teams • To improve communication among NGIs, among users and with users • Bring in expertise and services from Heavy User Communities (HUCs) Tools and services (current list – will be expanded): 1. EGI Application Database (Marios Chatziangelou, GRNET) 2. Services for VOs (Ignacio Blanquer Espert, UPV; Gonçalo Borges, LIP) 3. Training Marketplace (Claire Devereux, STFC) 4. Requirement Tracker (EGI.eu with help from CESNET)
18. www.egi.eu EGI-InSPIRE RI-261323 EGI Application Database WeNMR Devs Workshop, Utrecht, NL 20110712/13 18 Benefits of AppDB: • gives recognition to reusable applications and tools • gives recognition to application developers Use of AppDB • Register applications/tools • Reuse of applications/tools • Map of application landscape • AppDB gadget has been instantiated by: – WeNMR, LSGC, GridPP, Serbian, UK and Hungarian NGIs http://appdb.egi.eu/
19. www.egi.eu EGI-InSPIRE RI-261323 Coordination of services for VOs WeNMR Devs Workshop, Utrecht, NL 20110712/13 19 Coordination consists of: • Reviewing available services: management, monitor & accounting tools (NAGIOS SAM, GANGA, DIANE, CERN dashboard & mini dashboard) • Consultancy/help for VO managers(*) • Specification of requirements for improving operational tools to simplify VO management • Provisioning basic services for new VOs • Developing dashboard infrastructure (*) Further consultancy is provided by NGIs NGI NGI VO monitor NGI NGI Site resources https://wiki.egi.eu/wiki/VO_Services
20. www.egi.eu EGI-InSPIRE RI-261323 VO Services: NAGIOS SAM WeNMR Devs Workshop, Utrecht, NL 20110712/13 20 Service Status Details for: phys.vo.ibergrid.eu NAGIOS SAM triggers the executions of probes in grid sites • Tests to the Storage resources • Tests to the Computing Resources • Summary views of successful and unsuccessful services • Sends emails to the site admins in case of failures Provision of VO support software for VRCs & VOs (e.g. WeNMR, BioMed) Benefits: • Sites from a VO are tested (infrastructure tests run probes on NGIs) • Ability to develop and integrate VO’s own probes • A VO can use the central installation from LIP/UPV, or can install its own
21. www.egi.eu EGI-InSPIRE RI-261323 VO Services: VO Dashboard WeNMR Devs Workshop, Utrecht, NL 20110712/13 21 http://go.egi.eu/VO_Admin_Dashboard
22. www.egi.eu EGI-InSPIRE RI-261323 Training marketplace • Integration of: – Training event calendar – Digital library (of training materials) – Service offering & requesting form • Purpose of Training Marketplace: – Register training events – Use, reuse and share training materials – Offer training-related services – Feed training-requirements – Delegate representative to EGI Training Working Group • Results: – Approximately 60 training events were added during Project Year 1. WeNMR Devs Workshop, Utrecht, NL 20110712/13 22 http://go.egi.eu/Training_Marketplace
23. www.egi.eu EGI-InSPIRE RI-261323 Processing EGI requirements WeNMR Devs Workshop, Utrecht, NL 20110712/13 23 TCB EGI Requirements Tracker VRCs Input Channels for User requirements NGIs HUCs ESFRIs EEF VOs Developer teams in TNA3.4 UCST (with NGIs) UCB External technology providers EIROs EGI Helpdesk (GGUS system) • EMI • IGE • SAGA USAG • Permanent storage for requirements • Complete transparency – For user, NGIs & technology providers • Requirement submission interface http://go.egi.eu/wrt
24. www.egi.eu EGI-InSPIRE RI-261323 WeNMR RT Dashboard http://go.egi.eu/RT.WeNMR WeNMR Devs Workshop, Utrecht, NL 20110712/13 24
25. www.egi.eu EGI-InSPIRE RI-261323 PY1 Results: Requirements Tracker • 110 new requirements: – 48 resolved within EGI-InSPIRE/UCST – 36 prioritised by UCB of which… – 20 passed via TCB to Technology Providers – 26 still being investigated by UCST • Current issues endorsed by UCB and recognised by TCB: 1. Increased stability and scalability for gLite WMS 2. Better (more verbose and informative) error messages 3. Fixing the known bugs before adding new features 4. Coherency of command line commands, parameters and APIs 5. Better feedback about jobs, automated resubmission of jobs stuck on sites • Requirements to be refined in community workshops • Data management, Workflow systems & Workload management • Use to define and build EGI VRC User support platform WeNMR Devs Workshop, Utrecht, NL 20110712/13 25 Proved to be an effective communication channel with EMI and other external Technology Providers 60 requirements also generated from legacy documents from VRCs: SHARE Roadmap (HealthGrid), ESFRI project reqs., Production Grid Infrastucture (OGF), Request for Enhancements of gLite to support bio-NMR applications, gLite requirements (EGEE), User requirements (EGI-DS), DRIHMS Questionnaire
26. www.egi.eu EGI-InSPIRE RI-261323 Community Support Workshops • Motivation for workshops (from EGI Forums): – Users indicated need for – collaboration & exploration – More smaller and focused meetings needed – Bring specialists together • Purpose: – Inform and educate, clarify and specify • Emerging themes (from requirements): – portals, visualisation, data management, scalability, workflow, supporting users, planning training events, developing and provisioning sustainable apps. WeNMR Devs Workshop, Utrecht, NL 20110712/13 26 http://www.doodle.com/z8xp9hvnas4k64ch
27. www.egi.eu EGI-InSPIRE RI-261323 www.egi.eu/user-support WeNMR Devs Workshop, Utrecht, NL 20110712/13 27
28. www.egi.eu EGI-InSPIRE RI-261323 WeNMR as a VRC WeNMR signed an MoU with EGI 11 April 2011 WeNMR Devs Workshop, Utrecht, NL 20110712/13 28 http://www.egi.eu/collaboration/WeNMR.html
29. www.egi.eu EGI-InSPIRE RI-261323 Summary • EGI VRC model available and WeNMR VRC created – WeNMR participating in User Community Board – WeNMR contributing to Requirements Tracker process – EGI supporting NGIs who support WeNMR • Technical Support Services:  User requirements https://rt.egi.eu/rt/Dashboards/1054/WeNMR  AppDB gadget http://wenmr.eu/wenmr/wenmr-grid-statistics  VO services https://grid-monitor03.pd.infn.it:50080/nagios  Training Marketplace http://go.egi.eu/WeNMR-Workshop-20110819 • Community Support Workshops: – Themes under investigation WeNMR Devs Workshop, Utrecht, NL 20110712/13 29