Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Grid Monitoring at CERN with Elastic

Dd9d954997353b37b4c2684f478192d3?s=47 Elastic Co
November 10, 2015

Grid Monitoring at CERN with Elastic

In this talk Pablo Saiz presents the five different use cases where Elastic is used for WLCG monitoring at CERN:
- Messaging
- Job Monitoring
- Data Monitoring
- Infrastructure Monitoring
- Cloud Benchmarking
Pablo will also highlight the future goals with Elastic and their plan to expand their Elastic useage at CERN.

Pablo Saiz | Elastic{ON}Tour Munich | November 10, 2015

Dd9d954997353b37b4c2684f478192d3?s=128

Elastic Co

November 10, 2015
Tweet

More Decks by Elastic Co

Other Decks in Technology

Transcript

  1. Pablo Saiz 6-November-2015 1 Grid Monitoring at CERN with Elastic

  2. IT-SDC IT-SDC Elastic{ON}Tour in Munich, Pablo Saiz, CERN Table of

    contents ▪ CERN ▪ Worldwide LHC Computing GRID (WLCG) ▪ Elastic for WLCG Monitoring ▪ Messaging ▪ Job Monitoring ▪ Data Monitoring ▪ Infrastructure monitoring ▪ Cloud benchmarking 05/11/15 2
  3. IT-SDC 3 CERN was founded 1954: 12 European States “Science

    for Peace” Today: 21 Member States Member States: Austria, Belgium, Bulgaria, the Czech Republic, Denmark, Finland, France, Germany, Greece, Hungary, Israel, Italy, the Netherlands, Norway, Poland, Portugal, Slovakia, Spain, Sweden, Switzerland and the United Kingdom Candidate for Accession: Romania Associate Member in Pre-Stage to Membership: Serbia Applicant States for Membership or Associate Membership:
 Brazil, Cyprus, Pakistan, Russia, Slovenia, Turkey, Ukraine Observers to Council: India, Japan, Russia, Turkey, United States of America; European Commission and UNESCO ~ 2300 staff ~ 1600 other paid personnel ~ 10500 users Budget (2014) ~1000 MCHF Well known for: • Physics • WWW • Media: Angels & Demons, Flashforward, Daily show
  4. IT-SDC IT-SDC Elastic{ON}Tour in Munich, Pablo Saiz, CERN CERN’s Accelerator

    Complex 05/11/15 4 http://cernland.net
  5. IT-SDC IT-SDC Elastic{ON}Tour in Munich, Pablo Saiz, CERN Colliding particles

    (event) 05/11/15 5 Black Hole Higgs
  6. IT-SDC IT-SDC Elastic{ON}Tour in Munich, Pablo Saiz, CERN WLCG ▪

    Biggest scientific Grid project in the world ▪ ~170 computer centers (site) ▪ 1 Tier 0 (distributed in two locations) ▪ 12 bigger centers (Tier 1) ▪ ~160 Tier 2 ▪ 42 countries ▪ 10,000 users ▪ Running since Oct 2008 ▪ 2 million jobs per day ▪ ~600.000 cores ▪ 300 PB data 05/11/15 6 http://cern.ch/wlcg
  7. IT-SDC IT-SDC Elastic{ON}Tour in Munich, Pablo Saiz, CERN WLCG 05/11/15

    7 Including Max-Planck- Institut für Physik , Munich
  8. IT-SDC IT-SDC Elastic{ON}Tour in Munich, Pablo Saiz, CERN WLCG Monitoring

    ▪ Many different tools: ▪ Job, data transfers, services ▪ Usual Architecture: ▪ Python, RDBMS, apache, javascript ▪ Running since 2007 ▪ Team of ~10 people ▪ Different audience: ▪ Site/service/experiment managers, end users, general public 09/11/15 8 http://dashboard.cern.ch
  9. IT-SDC Elastic{ON} Tour in Munich, Pablo Saiz, CERN ElasticSearch configuration

    ▪ 2 clusters ▪ Production: 8 data, 2 search, 3 master ▪ Development: 2 data, 1 search, 1 master ▪ Data nodes: physical, 32GB, 32 core ▪ Search, master: virtual 8GB ▪ ~10 different use cases
  10. IT-SDC IT-SDC Elastic{ON}Tour in Munich, Pablo Saiz, CERN Use case

    1: Messaging Team (MIG) ▪ Goal: ▪ Check the status of the machines and services used for Messaging ▪ 25 machines, 15 clusters, ~130 applications, 260 M msg/day ▪ Using ElasticSearch and Kibana 3 in production since Aug 2014 ▪ 3 daily indexes, ~3M documents (850MB) per day ▪ Documents contain sensitive information ▪ Ensure only authorized people can access them 05/11/15 10 Messaging infrastructure ElasticSearch ESPER Transport
 layer (Messaging) Log Check Metric Status Graphite http://cern.ch/messaging
  11. IT-SDC IT-SDC Elastic{ON}Tour in Munich, Pablo Saiz, CERN MIG Dashboard

    05/11/15 11
  12. IT-SDC IT-SDC Elastic{ON}Tour in Munich, Pablo Saiz, CERN Use case

    2: Job Monitoring ▪ Goal: follow up the status of job processing ▪ ~2M daily jobs (170M updates) over ~170 sites ▪ Multiple applications: interactive view, summaries, accounting… ▪ Landing page query with 2 million records aggregation ▪ Aggregation 60M entries (100GB) ▪ In production (with RDBMS) since 2008 ▪ Elasticsearch alternative in place ▪ Currently checking data consistency 05/11/15 12
  13. IT-SDC IT-SDC Elastic{ON}Tour in Munich, Pablo Saiz, CERN Accessing raw

    data / summary 05/11/15 13
  14. IT-SDC IT-SDC Elastic{ON}Tour in Munich, Pablo Saiz, CERN Architecture ▪

    Python collectors ▪ Apache web server ▪ With mod_python ▪ Javascript ▪ Datatables, highcarts… ▪ Currently replacing the storage with ES 05/11/15 14
  15. IT-SDC IT-SDC Elastic{ON}Tour in Munich, Pablo Saiz, CERN ElasticSearch result

    ▪ Good speed-up in job parsing: ▪ From 220 jobs per second to 580 (2,6x) ▪ Better speed-up in applications: ▪ From 12 seconds to 0,70s in the most used query ▪ More queries possibilities: ▪ More than one month of aggregation 05/11/15 15
  16. IT-SDC IT-SDC Elastic{ON}Tour in Munich, Pablo Saiz, CERN Performance 05/11/15

    16 Opening the landing page lasticSearch S (no cache) 0 3 6 9 12 11.8 0.6 Maximum updates per second Python +ElasticSearch Python +RDBMS 0 150 300 450 600 223 584 One Month Query lasticSearch S (no cache) 0 300 600 900 1200 1084.3 14.1
  17. IT-SDC IT-SDC Elastic{ON}Tour in Munich, Pablo Saiz, CERN Use case

    3: Infrastructure Monitoring ▪ Goal: ▪ Follow up status of sites/services using user defined metrics ▪ 2M entries per day (1M raw metrics, 1 M combined metrics) ▪ Metrics: ▪ Numerical or string ▪ Non-linear in time (ndependent sources, future value, changing old entries) ▪ Keeping only status changes ▪ Combining metrics (AND/OR/ANY/ALL/FILTER) ▪ Creating ‘Views’ as lists of metrics ▪ Use in production (RDBMS) since 2010 ▪ ES alternative ready for data insertion and combining metrics ▪ Moving from python to java ▪ Using JEST interface ▪ Using ESPER ▪ ESPER/ES reduce significantly latency of metric combination ▪ UI needs to be modified 05/11/15 17 http://wlcg-mon.cern.ch/
  18. IT-SDC IT-SDC WLCG Monitoring Consolidation, Pablo Saiz, CERN Site Status

    Board 05/11/15 18
  19. IT-SDC IT-SDC Elastic{ON}Tour in Munich, Pablo Saiz, CERN How to

    combine metrics 09/11/15 19 Submit: ce1, ce2 OK ce3: FAIL Topo: Site1: ce1, ce2, St1, Tr1 Sote2: ce3, St2 Downtime ce3: maint CE ce1: OK ce2, ce3:FAIL Storage St1: OK St2: FAIL Transfer Tr1:OK CE Site1: Ok Site2: MAINT Storage Site1: OK Site2: FAIL Transfer Site1: OK Storage+ Transfer Site1:OK Site2: FAIL Status Site1: OK Site2: FAIL HOST SITE Example: Any CE AND ( Storage OR Transfer) Where CE=Submit AND Efficiency>80; Storage=Read AND Write; Transfer=CanTransfer Downtimes; Topology Efficiency: ce1:85 ce2:60 ce3:90 Read: St1: Ok St2: OK Write: St1 OK St2: FAIL Transfer: Tr1 OK CE ce1: OK Ce2: FAIL Ce3:MAINT Storage St1: OK St2: FAIL Transfer Tr1:OK
  20. IT-SDC IT-SDC Elastic{ON}Tour in Munich, Pablo Saiz, CERN Use case

    4: Data Monitoring ▪ Goal: ▪ Provide speed, success rate, volume of data movements ▪ 25M daily transfers, avg speed 10 GB/s ▪ RDBMS version in production since 2010 ▪ Elastic alternative ▪ 1 index: 160M documents, 80 GB ▪ Current status: ES index created, and being populated. Still to work on UI 05/11/15 20
  21. IT-SDC IT-SDC Elastic{ON}Tour in Munich, Pablo Saiz, CERN WLCG Transfers

    dashboard 05/11/15 21
  22. IT-SDC IT-SDC Elastic{ON}Tour in Munich, Pablo Saiz, CERN Use case

    5: Benchmarking Commercial Cloud Resources ▪ Goal: Compare different benchmark tools on different cloud providers ▪ Use 6 different benchmark tools and X cloud providers ▪ 1 index: 270K documents (~250 MB) ▪ Detailed analysis performed with Ipython analytics http://indico.cern.ch/event/384358/session/12/contribution/25 05/11/15 22
  23. IT-SDC IT-SDC Elastic{ON}Tour in Munich, Pablo Saiz, CERN Benchmark results

    at a glance ▪ Metric ~ [s]; each point ⬄10 min average; Colour ⬄ Cloud 05/11/15 23
  24. IT-SDC IT-SDC Elastic{ON}Tour in Munich, Pablo Saiz, CERN Qualitative look

    at data 05/11/15 24 1 VM 16 VMs 30 VMs 16 VMs 25 VMs 20 VMs ▪ Identifiable transition of CPU performance when load changes ▪ Seen in all benchmark measurements. Performance recovers scaling down
  25. IT-SDC IT-SDC Elastic{ON}Tour in Munich, Pablo Saiz, CERN Qualitative look

    at data ▪ Larger dispersion in KV and FastBmk values in the highest- load region 05/11/15 25 1 VM 16 VMs 30 VMs 16 VMs 25 VMs 20 VMs
  26. IT-SDC IT-SDC Analysis done outside elastic ▪ Correlation between different

    fields in documents ▪ Histograms with floating numbers 05/11/15 26 Evolution of a single VM in the parameter space FastBmk Vs KV Ratio mean(30VMs)/mean(16 VMs) Projection-X Aggr. x hypervisor Projection-Y Aggr. x hypervisor Profile-X 2D plot KV FastBmk A single 
 Hyperv.!!
  27. IT-SDC IT-SDC Elastic{ON}Tour in Munich, Pablo Saiz, CERN Next goals

    ▪ Use ES for Job Monitoring in production ▪ Migrate to ES 2 ▪ Check indexes/shards ▪ Ensure fine-grained security ▪ Currently, based on apache configuration (allow only REST interface!) ▪ Need to move to another alternative ▪ Move to Kibana 4 (once security in place) 05/11/15 27
  28. IT-SDC IT-SDC Elastic{ON}Tour in Munich, Pablo Saiz, CERN Summary ▪

    WLCG Grid Monitoring at CERN has been using Elastic in production for over year ▪ Currently, evaluating ElasticSearch for more use cases ▪ Very promising results so far! ▪ Looking forwards to expanding our Elastic usage! 05/11/15 28
  29. www.elastic.co