No content

Monitoring CERN Data Center and WLCG Experiments Monitoring CERN DC and WLCG 2 Diogo Nicolau – CERN/IT

Monitoring CERN DC and WLCG 3

Monitoring CERN DC and WLCG 4 About myself • Software Engineer • Joined CERN not so long ago • Designing and operating complex data pipelines dlimanic@cern.ch

Monitoring CERN DC and WLCG 5 How did I end up Monitoring “The crazy adventures of someone who just wanted to properly deploy a Recommendations System” Image: © 2015 – Susan Rossell

Monitoring CERN DC and WLCG 6 How did I end up Monitoring Me building ML models Me getting monitoring data

Monitoring CERN DC and WLCG 7 • Over 300 people • Enable the laboratory to fulfill its mission • Data Center and more: The IT Department IT Services Experiments Services Engineering Infrastructure Batch Storage Network Web Servers SW builds Chip design Hotel Bikes

Monitoring CERN DC and WLCG 8 CERN Data Centre: Primary Copy of LHC Data 70k disks 13k servers More than 300PB on tapes More than More than 300 000 300 000 cores cores Private Openstack Cloud

Tier-0 (CERN)  Data distribution  Data recording & archiving  20-40 Gbit/s connect to Tier1s  Tier-1s (13 centres)  Initial data reconstruction  Permanent storage  Re-processing  Tier-2s (>150 centres) Simulation End-user analysis WLCG: LHC Computing Grid Image credit: CERN 170 sites WORLDWIDE > 10k users 250k jobs CONCURRENTLY > 600k cores 15% of CERN RESOURCES > 700 PB storage

Monitoring CERN DC and WLCG 10 Monitoring Mission • Provide Monitoring as a Service for CERN Data Center (DC), IT Services and the WLCG collaboration • e.g. Dashboards, Alarms, Search, Archive • Collect, transport, store and process metrics and logs for applications and infrastructure

Monitoring CERN DC and WLCG 11

Monitoring CERN DC and WLCG 12 Challenges Rate and Volume From ~ 40k machines More than 3 TB/day (compressed)

Monitoring CERN DC and WLCG 13 Challenges Variety and Reliability More than 150 producers

1 modular architecture / built on open source tools 2 easy data integration / multiple ingestion endpoints 3 decoupled producers & consumers 4 built-in stream processing 5 support multiple backends with different SLAs Monitoring CERN DC and WLCG 14 Key Concepts

Monitoring CERN DC and WLCG 15

Monitoring CERN DC and WLCG 16 Observability Pipeline? @lucamag @smithclay

Monitoring CERN DC and WLCG 17

Data Center Base monitoring all nodes  40k nodes running Collectd  OS and HW metrics and alarms Service specific monitoring  Custom or upstream plugins  Monit metrics endpoint

Service Availability Historical View  Availability per service  Outages integration

20 WLCG Experiments Transfers classified by location, country, site… “Shifters” who keep tight control on the dasboard

Job Monitoring Jobs classified by state, source, resource…

Alarming Local (on the machine)  Simple Threshold / Actuators On dashboards  Grafana alert engine External  Alarm source Integrated with ticketing system  Service now

Service Alarms Service Plugins create the required metrics Include Puppet module and overwrite configurations

Monitoring CERN DC and WLCG 24

Monitoring CERN DC and WLCG 25 The team 2 Portuguese + 2 Spaniards + 1 Italian + 1 Bulgarian We work as any Dev team:  2 weeks sprints  15min dailys  1 member in “rota” Teams reach us by:  Mattermost  Mail  SnowTicket We mantain our code in git:  Puppet  Repo per Spark job  Gitlab CI works a charm

Stable production infrastructure integrated many different monitoring use cases can scale to its continuous growth Monitoring CERN DC and WLCG 26 Monitoring by Numbers: ~ 900 Active Users > 1000 Dashboards ~ 1000000 Queries/day > 30 Grafana Orgs tapping monit datasources

No content