Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Elk stack for analysing performance test results

Elk stack for analysing performance test results

How can a big data tool be used to analyse performance test results to understand trends and also create alerts.

More Decks by Anoop Vijayan Maniankara

Other Decks in Technology

Transcript

  1. 2008–2017 © Tuxera Inc. All Rights Reserved. Analysing performance test

    results with ELK Anoop Vijayan Maniankara Tuxera Inc. @maniankara 2 2008–2017 © Tuxera Inc. All Rights Reserved.
  2. 2008–2017 © Tuxera Inc. All Rights Reserved. Elk (Elastic) stack

    5.x • Elastic search – Logs storing (~NoSQL database) • Log stash – Log processor • Kibana – Graphs and charts over JSON Data shippers • Filebeat • Metricbeat • Packet beat • …
  3. 2008–2017 © Tuxera Inc. All Rights Reserved. Anoop Vijayan 12+

    years experience in designing, developing, testing Continuous Integration systems in bigger enterprises to small companies DevOps lead at Tuxera Inc. @maniankara Github, stackoverflow, etc.
  4. 2008–2017 © Tuxera Inc. All Rights Reserved. Agenda • All

    about logs –Purpose of log handling, tools available –With elkstack • Live log monitoring or processing • Static log/data analysis –Demo
  5. 2008–2017 © Tuxera Inc. All Rights Reserved. Purpose - Log

    handling Time Activities Longer Time • Search – Trace events from past • Analyze – Tracking trends over time Event • Monitor – Detect events • Compare – Two or more instances
  6. 2008–2017 © Tuxera Inc. All Rights Reserved. Tools in the

    market • Free and opensource: – ELK stack/Elastic stack - More manual configurations • Commercial (popular ones, not all) – Splunk – 1GB/day $4.5k (Perpetual)/$1.8k (Annual) – Loggly – Saas not available in-house $1.18k (1GB/day)/$2.38k (4GB/day) – …
  7. 2008–2017 © Tuxera Inc. All Rights Reserved. Architecture – ELK

    stack Souce: http://blog.arungupta.me/getting-started-elk-stack-wildfly/ L E K
  8. 2008–2017 © Tuxera Inc. All Rights Reserved. ELK stack (Kibana)

    look and feel Monitoring system logs Analyzing http request resources
  9. 2008–2017 © Tuxera Inc. All Rights Reserved. ELK stack (Kibana)

    look and feel contd. Analyzing event fields Comparing tools
  10. 2008–2017 © Tuxera Inc. All Rights Reserved. Scope • Live

    log handling • Static data comparison
  11. 2008–2017 © Tuxera Inc. All Rights Reserved. Live log parsing

    (nginx log) … 192.168.75.130 - - [23/Apr/2017:10:03:34 +0300] "GET /elasticsearch/_site/vendor.js HTTP/1.1" 304 0 "http://sandbox.srv.tuxera.com:3005/elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36" … … {"@timestamp":"2017-04- 23T07:55:37.886Z","beat":{"hostname":"sandbox.srv.tuxera.com","name":"sandbox.srv.tuxera.com","version":"5.2.2"},"input_type":"log","message":"192.168.75. 130 - - [23/Apr/2017:10:03:34 +0300] \"GET /elasticsearch/_site/vendor.js HTTP/1.1\" 304 0 \"http://sandbox.srv.tuxera.com:3005/elasticsearch/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36\"","offset":1595,"source":"/var/log/nginx/access.log","type":"log"} … Filebeat Log collection Logstash Log processing elasticsearch Log storing Kibana Log visualization access.log ELK stack … {”client”:” 192.168.75.130 ”,”time”:”23/Apr/2017:10:03:34 +0300”, ”request”:”GET”,”path”:” /elasticsearch/_site/vendor.js”,”response”:”304”,”host”:” sandbox.srv.tuxera.com”,”client”:”\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36\"”, ”beat”:{"hostname" : " sandbox.srv.tuxera.com","name" : " sandbox.srv.tuxera.com ","version" : "5.2.2”, "offset":1595,"source":"/var/log/nginx/access.log","type":"log"}} … … { "_index" : ”logstash-2017-04-23", "_type" : ”log", "_id" : "AVmDwg6kHxnroGLuXlMG", "_version" : 1, "found": true, "_source" : {”client”:” 192.168.75.130 ”,”time”:”23/Apr/2017:10:03:34 +0300”, ”request”:”GET”,”path”:” /elasticsearch/_site/vendor.js”,”response”:”304”,”host”:” sandbox.srv.tuxera.com”,”client”:”\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36\"”, ”beat”:{"hostname" : " sandbox.srv.tuxera.com","name" : " sandbox.srv.tuxera.com ","version" : "5.2.2”, "offset":1595,"source":"/var/log/nginx/access.log","type":"log"}}}
  12. 2008–2017 © Tuxera Inc. All Rights Reserved. Static data comparison

    Filebeat Log collection Logstash Log processing elasticsearch Log storing Kibana Log visualization Syslog ELK stack … {”time”:” Apr 23 06:47:01”, ”host”:” sandbox”, ”application”:”CRON”, ”processid”:” 23259”, ”cmd”:” (test -x /usr/sbin/anacron || ( cd / \u0026\u0026 run- parts --report /etc/cron.weekly ))”, ”beat”:{"hostname" : " sandbox.srv.tuxera.com","name" : " sandbox.srv.tuxera.com ","version" : "5.2.2"}} … … { "_index" : ”logstash-2017-04-23", "_type" : ”log", "_id" : "AVmDwg6kHxnroGLuXlMG", "_version" : 1, "found": true, "_source" : {”time”:” Apr 23 06:47:01”, ”host”:” sandbox”, ”application”:”CRON”, ”processid”:” 23259”, ”cmd”:” (test -x /usr/sbin/anacron || ( cd / \u0026\u0026 run-parts --report /etc/cron.weekly ))”, ”beat”:{"hostname" : " sandbox.srv.tuxera.com","name" : " sandbox.srv.tuxera.com ","version" : "5.2.2"}}} … {”time”:” Apr 23 06:47:01”, ”host”:” sandbox”, ”application”:”CRON”, ”processid”:” 23259”, ”cmd”:” (test -x /usr/sbin/anacron || ( cd / \u0026\u0026 run- parts --report /etc/cron.weekly ))”, ”beat”:{"hostname" : " sandbox.srv.tuxera.com","name" : " sandbox.srv.tuxera.com ","version" : "5.2.2"}} …
  13. 2008–2017 © Tuxera Inc. All Rights Reserved. Demos • Github

    path: https://github.com/maniankara/elk5.x-perf-testing
  14. 2008–2017 © Tuxera Inc. All Rights Reserved. Demo – 1

    – Live monitoring of nginx access.log • Demo1 – Live events – Filtering based on string Filebeat Log collection Logstash Log processing elasticsearch Log storing Kibana Log visualization access.log ELK stack • Recap
  15. 2008–2017 © Tuxera Inc. All Rights Reserved. Demo – 2

    – Helsinki air quality stats • Helsinki metropolitan air quality statistics (from HSY)
  16. 2008–2017 © Tuxera Inc. All Rights Reserved. Demo – 3

    –Tuxera performance tests • Terms to know • Effective read/write • Cache read/write –Performance improvement/degradation over time –commit comparison
  17. 2008–2017 © Tuxera Inc. All Rights Reserved. Thank you !

    Slides: https://goo.gl/ZZi5EL Demo: https://goo.gl/4JYBE9 Anoop Vijayan @maniankara