[2020.02 Meetup] [Talk #2] Diogo Nicolau - Monitoring CERN Data Centre and WLCG Experiments

This talk discusses the monitoring architecture, the challenges encountered in operating and scaling a pipeline to handle billions of events and presents how users benefit from a central monitoring service for processing and analysis of monitoring data.

Diogo is a Software Engineer with experience in designing and operating complex data pipelines. He earned his Masters in Computer Science at Instituto Superior Técnico with focus on intelligent decision systems. While yet studying he had the chance to explore several startups where he helped building from recommender systems to marketing newsletters. He joined the CERN IT Department to work on the evolution of monitoring infrastructure towards modern and scalable pipelines for data ingestion and stream processing. In his spare time, as a true millennial, he enjoys travelling and finding new places to eat as much as watching Netflix.

Transcript

2. Monitoring CERN Data Center and WLCG Experiments Monitoring CERN DC and WLCG 2 Diogo Nicolau – CERN/IT
3. Monitoring CERN DC and WLCG 3
4. Monitoring CERN DC and WLCG 4 About myself • Software Engineer • Joined CERN not so long ago • Designing and operating complex data pipelines dlimanic@cern.ch
5. Monitoring CERN DC and WLCG 5 How did I end up Monitoring “The crazy adventures of someone who just wanted to properly deploy a Recommendations System” Image: © 2015 – Susan Rossell
6. Monitoring CERN DC and WLCG 6 How did I end up Monitoring Me building ML models Me getting monitoring data
7. Monitoring CERN DC and WLCG 7 • Over 300 people • Enable the laboratory to fulfill its mission • Data Center and more: The IT Department IT Services Experiments Services Engineering Infrastructure Batch Storage Network Web Servers SW builds Chip design Hotel Bikes
8. Monitoring CERN DC and WLCG 8 CERN Data Centre: Primary Copy of LHC Data 70k disks 13k servers More than 300PB on tapes More than More than 300 000 300 000 cores cores Private Openstack Cloud
9. Tier-0 (CERN)  Data distribution  Data recording & archiving  20-40 Gbit/s connect to Tier1s  Tier-1s (13 centres)  Initial data reconstruction  Permanent storage  Re-processing  Tier-2s (>150 centres) Simulation End-user analysis WLCG: LHC Computing Grid Image credit: CERN 170 sites WORLDWIDE > 10k users 250k jobs CONCURRENTLY > 600k cores 15% of CERN RESOURCES > 700 PB storage
10. Monitoring CERN DC and WLCG 10 Monitoring Mission • Provide Monitoring as a Service for CERN Data Center (DC), IT Services and the WLCG collaboration • e.g. Dashboards, Alarms, Search, Archive • Collect, transport, store and process metrics and logs for applications and infrastructure
11. Monitoring CERN DC and WLCG 11
12. Monitoring CERN DC and WLCG 12 Challenges Rate and Volume From ~ 40k machines More than 3 TB/day (compressed)
13. Monitoring CERN DC and WLCG 13 Challenges Variety and Reliability More than 150 producers
14. 1 modular architecture / built on open source tools 2 easy data integration / multiple ingestion endpoints 3 decoupled producers & consumers 4 built-in stream processing 5 support multiple backends with different SLAs Monitoring CERN DC and WLCG 14 Key Concepts
15. Monitoring CERN DC and WLCG 15
16. Monitoring CERN DC and WLCG 16 Observability Pipeline? @lucamag @smithclay
17. Monitoring CERN DC and WLCG 17
18. Data Center Base monitoring all nodes  40k nodes running Collectd  OS and HW metrics and alarms Service specific monitoring  Custom or upstream plugins  Monit metrics endpoint
19. Service Availability Historical View  Availability per service  Outages integration
20. 20 WLCG Experiments Transfers classified by location, country, site… “Shifters” who keep tight control on the dasboard
21. Job Monitoring Jobs classified by state, source, resource…
22. Alarming Local (on the machine)  Simple Threshold / Actuators On dashboards  Grafana alert engine External  Alarm source Integrated with ticketing system  Service now
23. Service Alarms Service Plugins create the required metrics Include Puppet module and overwrite configurations
24. Monitoring CERN DC and WLCG 24
25. Monitoring CERN DC and WLCG 25 The team 2 Portuguese + 2 Spaniards + 1 Italian + 1 Bulgarian We work as any Dev team:  2 weeks sprints  15min dailys  1 member in “rota” Teams reach us by:  Mattermost  Mail  SnowTicket We mantain our code in git:  Puppet  Repo per Spark job  Gitlab CI works a charm
26. Stable production infrastructure integrated many different monitoring use cases can scale to its continuous growth Monitoring CERN DC and WLCG 26 Monitoring by Numbers: ~ 900 Active Users > 1000 Dashboards ~ 1000000 Queries/day > 30 Grafana Orgs tapping monit datasources