Learn how CERN uses the Elastic Stack for five different use cases for the Worldwide LHC Computing Grid: Messaging, Job Monitoring, Data Monitoring, Infrastructure Monitoring, and Cloud Benchmarking.
CERN and Worldwide LHC Computing GRID (WLCG) • Elastic for CERN IT • Data center monitoring • Job Monitoring • Service Monitoring • Data transfers • CERN IT Elastic Service Many slides taken from François Briard, International Relations
Peace” Today: 21 Member States Member States: Austria, Belgium, Bulgaria, the Czech Republic, Denmark, Finland, France, Germany, Greece, Hungary, Israel, Italy, the Netherlands, Norway, Poland, Portugal, Slovakia, Spain, Sweden, Switzerland and the United Kingdom Candidate for Accession: Romania Associate Member in Pre-Stage to Membership: Serbia Applicant States for Membership or Associate Membership: Brazil, Cyprus, Pakistan, Russia, Slovenia, Turkey, Ukraine Observers to Council: India, Japan, Russia, Turkey, United States of America; European Commission and UNESCO ~ 2300 staff ~ 1600 other paid personnel ~ 11000 users Budget ~1000 MCHF Well known for: • Physics • WWW • Media: Angels & Demons, Flashforward, Daily show
project in the world • ~170 computer centers (site) • 1 Tier 0 (distributed in two locations) • 14 bigger centers (Tier 1) • ~160 Tier 2 • 42 countries • 10,000 users • Running since Oct 2008 • 3 million jobs per day • ~600.000 cores • 300 PB data • Do you want to contribute? • http://lhcathome.web.cern.ch/
GRID (WLCG) • Elastic for CERN IT • Data center monitoring • Job Monitoring • Service Monitoring • Data transfers • CERN IT Elastic Service 2/25/16 Pablo Saiz ElasticON 2016
Monitoring for Data Center and WLCG • Many different tools: • Data center, Job, data transfers, services • Team of ~10 people • Different audience: • Site/service/experiment managers, end users, general public • Multiple use cases • Hosts/Job/Services/Transfers • Some use cases with sensitive data: • Cloud benchmarking/Syslog
Job Monitoring • Goal: • follow up the status of job processing • ~3M daily jobs (170M updates) over ~170 sites • Multiple applications: interactive view, summaries, accounting… • Landing page aggregates last day • Aggregation 60M entries (100GB) • In production (with RDBMS) since 2008 • Elasticsearch alternative in place • Interactive view for one experiment
Monitoring • Goal: • Follow up status of sites/services using user defined metrics • Metrics: • Numerical or string • Non-linear in time (independent sources, future value, changing old entries) • Keeping only status changes • Combining metrics (AND/OR/ANY/ALL/FILTER/OVERWRITE) • 2M entries per day (1M raw metrics, 1 M combined metrics) • Creating ‘Views’ as lists of metrics • Use in production (RDBMS) since 2010 • Elastic version being developed
OR AND AND IF DATA <m> OVERWRITE <n> IF <N> != THEN <N> ELSE <M> ANY <m>, <t> OR of all instances in <m> that have the same value in <t> ALL <m>, <t> AND IF DATA of all instance in <m> that have the same value in <t> FILTER <m>, <t>=‘v’ Take only the instances of <m> that have a value of ‘v’ in metric <t> * OK WARNING CRITICAL DOWNTIME UNKNOWN NO DATA * Will be converted to if there is at least one more metric
Monitoring • Goal: • Provide transfer/success rate, volume of data movements • 25M daily transfers, avg speed 10 GB/s • RDBMS version in production since 2010 • Elastic alternative • 1 index: 160M documents, 80 GB • Current status: ES index created, and being populated Still working on UI
LHC Computing GRID (WLCG) • Elastic for CERN IT • Data center monitoring • Job Monitoring • Service Monitoring • Data transfers • CERN IT Elastic Service 2/25/16 Pablo Saiz ElasticON 2016
• Maintained by different teams • Multiple versions of ES: 1.4 à2.X • Different security setup • Duplication of data • Difficult to correlate • Need to consolidate! • Create small team to support Elasticsearch at CERN 35 2/25/16 Pablo Saiz ElasticON 2016
part time) • And should become even smaller • Prototype Elasticsearch 2.2 cluster • Couple of clients invited as guinea-pigs • Goal: production ready service by Q4 2016 36 2/25/16 Pablo Saiz ElasticON 2016
client? • Physical/virtual nodes • IO performance or ease of management? • Security, ACL • Ensure privacy and multitenant • Accounting • Only space? Include usage? • Migrate data from current clusters • Automate (PaaS) • FOUND? Elasticcloud? 38 2/25/16 Pablo Saiz ElasticON 2016
production for a couple of years • Multiple use cases: • Data Center, WLCG Job/Service/Data,… • Currently setting up a centralized Elasticsearch Service • Plenty of interest from many different clients • Goal to provide a production service by Q4 2016 40 2/25/16 Pablo Saiz ElasticON 2016
elastic
rinchsan
maimyyym
yoshiitaka
rainerhahnekamp
akagire
koki_nishihara
tamaiyutaro
you
minorun365
akkie76
tahia910
inesmontani
jnunemaker
aarron
notwaldorf
garrettdimon
trishagee
smashingmag
addyosmani
jlugia
palkan
chrisshort
tammielis
kneath
![1 Pablo Saiz 18 February 2016 [email protected] Monitoring at CERN](https://files.speakerdeck.com/presentations/66ccac74d5d04bc49a2783d5510b280a/slide_0.jpg){kind=link}
