How to start using the ELK stack to centralize All The Things and gain visibility into your data

Slides for talk delivered at DevOps and Automation NJ Meetup at Ishi Systems, Jersey City, NJ on May 12, 2015.

Transcript

1. How to start using the ELK stack to centralize All The Things and gain visibility into your data Peter Kim, Solutions Architect DevOps and Automation NJ Meetup May 12, 2015
2. www.elastic.co { "name": "Peter Kim", "title": "Solutions Architect", "employer": "Elastic", "kids": [ {"name": "Rafael", "age": 4}, {"name": "Mateo", "age": 2} ], "city": "Hoboken", "state": "New Jersey" } Me 2
3. www.elastic.co Logging Problems 3
4. www.elastic.co Problem 1: No Consistency • Every application and device logs in its own special way. • Expert in each log format required to use the logs. • Difficult to search across because of this formatting problem. 4
5. www.elastic.co Problem 1: No Consistency 120707 0:40:34 4 Connect root@localhost on 4 Query select @@version_comment limit 1 120707 0:40:45 4 Query select * from mysql.user 120707 0:41:18 5 Query hello world 5
6. www.elastic.co Problem 1: No Consistency 120707 0:37:09 [Note] Plugin 'FEDERATED' is disabled. 120707 0:37:09 InnoDB: The InnoDB memory heap is disabled 120707 0:37:09 InnoDB: Mutexes and rw_locks use GCC atomic builtins 120707 0:37:09 InnoDB: Compressed tables use zlib 1.2.5 120707 0:37:09 InnoDB: Using Linux native AIO 120707 0:37:09 InnoDB: Initializing buffer pool, size = 128.0M 120707 0:37:09 InnoDB: Completed initialization of buffer pool 6
7. www.elastic.co Problem 1: No Consistency # User@Host: biz_1[biz_1] @ localhost [] # Query_time: 0.000273 Lock_time: 0.000104 Rows_sent: 1 Rows_examined: 1 SET timestamp=1255345490; SELECT * FROM organization_details; 7
8. www.elastic.co Problem 1: No Consistency Mar 23 22:05:24 Macintosh com.apple.launchd[1] (httpd): Throttling respawn: Will start in 10 seconds 8
9. www.elastic.co Problem 2: Datetime formats • [29/Apr/2011:07:05:26 +0000] • Oct 11 20:21:47 • 130460505 • 020805 13:51:24 9
10. www.elastic.co Problem 3: De-centralized • Logs are spread across all your servers • Many servers have many different kinds of logs • ssh + grep aren't scalable 10
11. www.elastic.co Problem 4: Experts required • People interested in the logs often… – Do not have access to read the logs – Do not have expertise to understand the data – Do not know where the logs are 11
12. 66.249.73.185 - - [16/Feb/2014:09:47:54 -0500] "GET / HTTP/1.1" 200 37932 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
13. 93.114.45.13 - - [16/Feb/2014:09:47:04 -0500] "GET /favicon.ico HTTP/1.1" 200 3638 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:25.0) Gecko/20100101 Firefox/25.0" 93.114.45.13 - - [16/Feb/2014:09:47:04 -0500] "GET /images/jordan-80.png HTTP/1.1" 200 6146 "http://www.semicomplete.com/articles/dynamic-dns-with- dhcp/" "Mozilla/5.0 (X11; Linux x86_64; rv:25.0) Gecko/20100101 Firefox/25.0" 93.114.45.13 - - [16/Feb/2014:09:47:04 -0500] "GET /images/web/2009/banner.png HTTP/1.1" 200 52315 "http://www.semicomplete.com/style2.css" "Mozilla/5.0 (X11; Linux x86_64; rv:25.0) Gecko/20100101 Firefox/25.0" 66.249.73.135 - - [16/Feb/2014:09:47:34 -0500] "GET /blog/tags/ipv6 HTTP/1.1" 200 12251 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 50.16.19.13 - - [16/Feb/2014:09:47:46 -0500] "GET /blog/tags/puppet?flav=rss20 HTTP/1.1" 200 14872 "http://www.semicomplete.com/blog/tags/puppet?flav=rss20" "Tiny Tiny RSS/1.11 (http://tt-rss.org/)" 66.249.73.185 - - [16/Feb/2014:09:47:54 -0500] "GET / HTTP/1.1" 200 37932 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 110.136.166.128 - - [16/Feb/2014:09:48:42 -0500] "GET /projects/xdotool/ HTTP/1.1" 200 12292 "http://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=5&cad=rja&sqi=2&ved=0CFYQFjAE&url=http%3A%2F%2Fwww.semicomplete.com% 2Fprojects%2Fxdotool%2F&ei=6cwAU_bRHo6urAeI0YD4Ag&usg=AFQjCNE3V_aCf3-gfNcbS924S6jZ6FqffA&bvm=bv.61535280,d.bmk" "Mozilla/5.0 (Windows NT 6.2; WOW64; rv:28.0) Gecko/20100101 Firefox/28.0" 46.105.14.53 - - [16/Feb/2014:09:48:48 -0500] "GET /blog/tags/puppet?flav=rss20 HTTP/1.1" 200 14872 "-" "UniversalFeedParser/4.2-pre-314-svn +http://feedparser.org/" 110.136.166.128 - - [16/Feb/2014:09:48:53 -0500] "GET /reset.css HTTP/1.1" 200 1015 "http://www.semicomplete.com/projects/xdotool/" "Mozilla/5.0 (Windows NT 6.2; WOW64; rv:28.0) Gecko/20100101 Firefox/28.0" 110.136.166.128 - - [16/Feb/2014:09:48:53 -0500] "GET /style2.css HTTP/1.1" 200 4877 "http://www.semicomplete.com/projects/xdotool/" "Mozilla/5.0 (Windows NT 6.2; WOW64; rv:28.0) Gecko/20100101 Firefox/28.0" 110.136.166.128 - - [16/Feb/2014:09:48:53 -0500] "GET /favicon.ico HTTP/1.1" 200 3638 "-" "Mozilla/5.0 (Windows NT 6.2; WOW64; rv:28.0) Gecko/20100101 Firefox/28.0" 110.136.166.128 - - [16/Feb/2014:09:48:53 -0500] "GET /images/jordan-80.png HTTP/1.1" 200 6146 "http://www.semicomplete.com/projects/xdotool/" "Mozilla/5.0 (Windows NT 6.2; WOW64; rv:28.0) Gecko/20100101 Firefox/28.0" 123.125.71.35 - - [16/Feb/2014:09:49:02 -0500] "GET /blog/tags/release HTTP/1.1" 200 40693 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)" 110.136.166.128 - - [16/Feb/2014:09:48:53 -0500] "GET /images/web/2009/banner.png HTTP/1.1" 200 52315 "http://www.semicomplete.com/style2.css" "Mozilla/5.0 (Windows NT 6.2; WOW64; rv:28.0) Gecko/20100101 Firefox/28.0"
14. 110.136.166.128 - - [16/Feb/2014:09:48:53 -0500] "GET /images/jordan-80.png HTTP/1.1" 200 6146 "http://www.semicomplete.com/projects/xdotool/" "Mozilla/5.0 (Windows NT 6.2; WOW64; rv:28.0) Gecko/20100101 Firefox/28.0" 123.125.71.35 - - [16/Feb/2014:09:49:02 -0500] "GET /blog/tags/release HTTP/1.1" 200 40693 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)" 110.136.166.128 - - [16/Feb/2014:09:48:53 -0500] "GET /images/web/2009/banner.png HTTP/1.1" 200 52315 "http://www.semicomplete.com/style2.css" "Mozilla/5.0 (Windows NT 6.2; WOW64; rv:28.0) Gecko/20100101 Firefox/28.0" 50.150.204.184 - - [16/Feb/2014:09:49:37 -0500] "GET /images/googledotcom.png HTTP/1.1" 200 65748 "http://www.google.com/search?q=https//:google.com&source=lnms&tbm=isch&sa=X&ei=4-r8UvDrKZOgkQe7x4CICw&ved=0CAkQ_AUoAA&biw=320&bih=441" "Mozilla/5.0 (Linux; U; Android 4.0.4; en-us; LG-MS770 Build/IMM76I) AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 Mobile Safari/534.30" 207.241.237.225 - - [16/Feb/2014:09:50:06 -0500] "GET /blog/tags/examples HTTP/1.0" 200 9208 "http://www.semicomplete.com/blog/tags/C" "Mozilla/5.0 (compatible; archive.org_bot +http://www.archive.org/details/archive.org_bot)" 200.49.190.101 - - [16/Feb/2014:09:50:10 -0500] "GET /reset.css HTTP/1.1" 200 1015 "-" "-" 200.49.190.100 - - [16/Feb/2014:09:50:08 -0500] "GET /blog/tags/web HTTP/1.1" 200 44019 "-" "QS304 Profile/MIDP-2.0 Configuration/CLDC-1.1" 200.49.190.101 - - [16/Feb/2014:09:50:12 -0500] "GET /style2.css HTTP/1.1" 200 4877 "-" "-" 200.49.190.101 - - [16/Feb/2014:09:50:19 -0500] "GET /images/jordan-80.png HTTP/1.1" 200 6146 "-" "QS304 Profile/MIDP-2.0 Configuration/CLDC-1.1" 66.249.73.185 - - [16/Feb/2014:09:51:19 -0500] "GET /reset.css HTTP/1.1" 200 1015 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 66.249.73.135 - - [16/Feb/2014:09:51:26 -0500] "GET /blog/tags/munin HTTP/1.1" 200 9746 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 66.249.73.135 - - [16/Feb/2014:09:51:47 -0500] "GET /blog/tags/firefox?flav=rss20 HTTP/1.1" 200 16021 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 66.249.73.135 - - [16/Feb/2014:09:52:34 -0500] "GET /blog/geekery/eventdb-ideas.html HTTP/1.1" 200 11418 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 67.214.178.190 - - [16/Feb/2014:09:53:19 -0500] "GET / HTTP/1.0" 200 37932 "http://www.semicomplete.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:21.0) Gecko/20100101 Firefox/21.0" 67.214.178.190 - - [16/Feb/2014:09:53:30 -0500] "GET /blog/geekery/installing-windows-8-consumer-preview.html HTTP/1.0" 200 8948 "http://www.semicomplete.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:21.0) Gecko/20100101 Firefox/21.0" 207.241.237.220 - - [16/Feb/2014:09:53:47 -0500] "GET /blog/tags/projects HTTP/1.0" 200 28370 "http://www.semicomplete.com/blog/tags/C" "Mozilla/5.0 (compatible; archive.org_bot +http://www.archive.org/details/archive.org_bot)" 46.105.14.53 - - [16/Feb/2014:09:53:48 -0500] "GET /blog/tags/puppet?flav=rss20 HTTP/1.1" 200 14872 "-" "UniversalFeedParser/4.2-pre-314-svn +http://feedparser.org/" 207.241.237.227 - - [16/Feb/2014:09:53:50 -0500] "GET /blog/geekery/soekris-gpio.html HTTP/1.0" 200 9587 "http://www.semicomplete.com/blog/tags/C" "Mozilla/5.0 (compatible; archive.org_bot +http://www.archive.org/details/archive.org_bot)" 91.177.205.119 - - [16/Feb/2014:09:54:34 -0500] "GET /blog/geekery/xvfb-firefox.html HTTP/1.1" 200 10975 "http://en.wikipedia.org/wiki/Xvfb" "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0)" 91.177.205.119 - - [16/Feb/2014:09:54:34 -0500] "GET /reset.css HTTP/1.1" 200 1015 "http://semicomplete.com/blog/geekery/xvfb-firefox.html" "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0)" 91.177.205.119 - - [16/Feb/2014:09:54:34 -0500] "GET /style2.css HTTP/1.1" 200 4877 "http://semicomplete.com/blog/geekery/xvfb-firefox.html" "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0)" 91.177.205.119 - - [16/Feb/2014:09:54:34 -0500] "GET /images/jordan-80.png HTTP/1.1" 200 6146 "http://semicomplete.com/blog/geekery/xvfb-firefox.html" "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0)" 91.177.205.119 - - [16/Feb/2014:09:54:34 -0500] "GET /images/web/2009/banner.png HTTP/1.1" 200 52315 "http://semicomplete.com/blog/geekery/xvfb-firefox.html" "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0)" 91.177.205.119 - - [16/Feb/2014:09:54:35 -0500] "GET /favicon.ico HTTP/1.1" 200 3638 "-" "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Win64; x64; Trident/6.0)" 66.249.73.185 - - [16/Feb/2014:09:54:44 -0500] "GET /doc/index.html?org/elasticsearch/action/search/SearchResponse.html HTTP/1.1" 404 294 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 207.241.237.228 - - [16/Feb/2014:09:54:54 -0500] "GET /blog/tags/defcon HTTP/1.0" 200 24142 "http://www.semicomplete.com/blog/tags/C" "Mozilla/5.0 (compatible; archive.org_bot +http://www.archive.org/details/archive.org_bot)" 207.241.237.101 - - [16/Feb/2014:09:54:58 -0500] "GET /blog/tags/regex HTTP/1.0" 200 14888 "http://www.semicomplete.com/blog/tags/C" "Mozilla/5.0 (compatible; archive.org_bot +http://www.archive.org/details/archive.org_bot)" 87.169.99.232 - - [16/Feb/2014:09:56:12 -0500] "GET /presentations/puppet-at-loggly/puppet-at-loggly.pdf.html HTTP/1.1" 200 24747 "https://www.google.de/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36" 209.85.238.199 - - [16/Feb/2014:09:56:18 -0500] "GET /blog/tags/firefox?flav=rss20 HTTP/1.1" 200 16021 "-" "Feedfetcher-Google; (+http://www.google.com/feedfetcher.html; 3 subscribers; feed-id=14171215010336145331)" 209.85.238.199 - - [16/Feb/2014:09:56:31 -0500] "GET /test.xml HTTP/1.1" 200 1370 "-" "Feedfetcher-Google; (+http://www.google.com/feedfetcher.html; 1 subscribers; feed-id=11390274670024826467)" 81.220.24.207 - - [16/Feb/2014:09:57:28 -0500] "GET /blog/geekery/ssl-latency.html HTTP/1.1" 200 17147 "http://www.google.fr/url?sa=t&rct=j&q=&esrc=s&source=web&cd=5&ved=0CE4QFjAE&url=http%3A%2F%2Fwww.semicomplete.com%2Fblog%2Fgeekery%2Fssl- latency.html&ei=ZdEAU9mGGuWX1AW09IDoBw&usg=AFQjCNHw6zioJpizqX8Q0YpKKaF4zdCSEg&bvm=bv.61535280,d.d2k" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.73.11 (KHTML, like Gecko) Version/7.0.1 Safari/537.73.11" 81.220.24.207 - - [16/Feb/2014:09:57:28 -0500] "GET /reset.css HTTP/1.1" 200 1015 "http://www.semicomplete.com/blog/geekery/ssl-latency.html" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.73.11 (KHTML, like Gecko) Version/7.0.1 Safari/537.73.11" 81.220.24.207 - - [16/Feb/2014:09:57:28 -0500] "GET /images/jordan-80.png HTTP/1.1" 200 6146 "http://www.semicomplete.com/blog/geekery/ssl-latency.html" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.73.11 (KHTML, like Gecko) Version/7.0.1 Safari/537.73.11" 81.220.24.207 - - [16/Feb/2014:09:57:28 -0500] "GET /style2.css HTTP/1.1" 200 4877 "http://www.semicomplete.com/blog/geekery/ssl-latency.html" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.73.11 (KHTML, like Gecko) Version/7.0.1 Safari/537.73.11" 81.220.24.207 - - [16/Feb/2014:09:57:28 -0500] "GET /images/web/2009/banner.png HTTP/1.1" 200 52315 "http://www.semicomplete.com/blog/geekery/ssl-latency.html" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.73.11 (KHTML, like Gecko) Version/7.0.1 Safari/537.73.11" 81.220.24.207 - - [16/Feb/2014:09:57:29 -0500] "GET /favicon.ico HTTP/1.1" 200 3638 "http://www.semicomplete.com/blog/geekery/ssl-latency.html" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.73.11 (KHTML, like Gecko) Version/7.0.1 Safari/537.73.11" 66.249.73.135 - - [16/Feb/2014:09:57:36 -0500] "GET /blog/geekery/vmware-cpu-performance.html HTTP/1.1" 200 12908 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 46.105.14.53 - - [16/Feb/2014:09:58:48 -0500] "GET /blog/tags/puppet?flav=rss20 HTTP/1.1" 200 14872 "-" "UniversalFeedParser/4.2-pre-314-svn +http://feedparser.org/" 218.30.103.62 - - [16/Feb/2014:09:59:36 -0500] "GET /robots.txt HTTP/1.1" 200 - "-" "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)" 218.30.103.62 - - [16/Feb/2014:09:59:41 -0500] "GET /robots.txt HTTP/1.1" 200 - "-" "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)" 218.30.103.62 - - [16/Feb/2014:09:59:46 -0500] "GET /projects/fex/ HTTP/1.1" 200 14352 "-" "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)" 74.125.40.20 - - [16/Feb/2014:09:59:53 -0500] "GET /?flav=rss20 HTTP/1.1" 200 29941 "-" "FeedBurner/1.0 (http://www.FeedBurner.com)" 71.212.224.97 - - [16/Feb/2014:10:00:05 -0500] "GET /projects/xdotool/ HTTP/1.1" 200 12292 "http://suckless.org/rocks" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36" 71.212.224.97 - - [16/Feb/2014:10:00:05 -0500] "GET /reset.css HTTP/1.1" 200 1015 "http://www.semicomplete.com/projects/xdotool/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36" 71.212.224.97 - - [16/Feb/2014:10:00:06 -0500] "GET /style2.css HTTP/1.1" 200 4877 "http://www.semicomplete.com/projects/xdotool/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36" 71.212.224.97 - - [16/Feb/2014:10:00:06 -0500] "GET /images/jordan-80.png HTTP/1.1" 200 6146 "http://www.semicomplete.com/projects/xdotool/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36" 71.212.224.97 - - [16/Feb/2014:10:00:06 -0500] "GET /images/web/2009/banner.png HTTP/1.1" 200 52315 "http://www.semicomplete.com/projects/xdotool/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36" 218.30.103.62 - - [16/Feb/2014:10:00:07 -0500] "GET /projects/xdotool/xdotool.xhtml HTTP/1.1" 304 - "-" "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)" 108.174.55.234 - - [16/Feb/2014:10:00:16 -0500] "GET /?flav=rss20 HTTP/1.1" 200 29941 "-" "-" 218.30.103.62 - - [16/Feb/2014:10:00:28 -0500] "GET /blog/geekery/c-vs-python-bdb.html HTTP/1.1" 200 11388 "-" "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)" 121.107.188.202 - - [16/Feb/2014:10:00:28 -0500] "GET /presentations/logstash-monitorama-2013/images/kibana-dashboard3.png HTTP/1.1" 200 171717 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36" 218.30.103.62 - - [16/Feb/2014:10:00:52 -0500] "GET /blog/productivity/better-zsh-xterm-title-fix.html HTTP/1.1" 200 10185 "-" "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)" 218.30.103.62 - - [16/Feb/2014:10:01:14 -0500] "GET /blog/geekery/xvfb-firefox.html HTTP/1.1" 200 10975 "-" "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)" 218.30.103.62 - - [16/Feb/2014:10:01:37 -0500] "GET /blog/geekery/puppet-facts-into-mcollective.html HTTP/1.1" 200 9872 "-" "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)" 198.46.149.143 - - [16/Feb/2014:10:01:44 -0500] "GET /blog/geekery/disabling-battery-in-ubuntu-vms.html?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+semicomplete%2Fmain+%28semicomplete.com+-+Jordan+Sissel%29 HTTP/1.1" 200 9316 "-" "Tiny Tiny RSS/1.11 (http://tt-rss.org/)" 198.46.149.143 - - [16/Feb/2014:10:01:44 -0500] "GET /blog/geekery/solving-good-or-bad-problems.html?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+semicomplete%2Fmain+%28semicomplete.com+-+Jordan+Sissel%29 HTTP/1.1" 200 10756 "-" "Tiny Tiny RSS/1.11 (http://tt-rss.org/)" 218.30.103.62 - - [16/Feb/2014:10:01:57 -0500] "GET /blog/geekery/jquery-interface-puffer.html%20target= HTTP/1.1" 200 202 "-" "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)" 218.30.103.62 - - [16/Feb/2014:10:02:19 -0500] "GET /blog/geekery/ec2-reserved-vs-ondemand.html HTTP/1.1" 200 11834 "-" "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)" 66.249.73.135 - - [16/Feb/2014:10:02:37 -0500] "GET /blog/web/firefox-scrolling-fix.html HTTP/1.1" 200 8956 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1;
15. www.elastic.co The Solution? The ELK stack! 15
16. www.elastic.co The ELK Stack • Elasticsearch + Logstash + Kibana • Open source Apache 2 license • Open source integrations (some examples): – Puppet, Chef, Ansible – Elasticsearch for Apache Hadoop • Commercial plugins: – Security – Alerting • Offered by a single company: Elastic 16
17. www.elastic.co Logstash: In 30 seconds • Managing events and logs • Collect, parse, enrich and store data • Modular: many, many inputs and outputs • Ruby app (JRuby) 17
18. www.elastic.co Logstash: Architecture Logstash Input Output Filter ? ? collect and split alter and enrich store and visualise 18
19. www.elastic.co • Monitoring: collectd, graphite, ganglia, snmptrap, zenoss • Datastores: elasticsearch, redis, sqlite, s3 • Queues: rabbitmq, zeromq • Logging: eventlog, lumberjack, gelf, log4j, relp, syslog, varnish log • Platforms: drupal_dblog, gemfire, heroku, sqs, s3, twitter • Local: exec, generator, file, stdin, pipe, unix • Protocol: imap, irc, stomp, tcp, udp, websocket, wmi, xmpp Logstash: Inputs 19
20. www.elastic.co • alter, anonymize, checksum, csv, drop, multiline • dns, date, extractnumbers, geoip, i18n, kv, noop, ruby, range • json, urldecode, useragent • metrics, sleep • grok • … many, many more … Logstash: Filters 20
21. www.elastic.co • Store: elasticsearch, gemfire, mongodb, redis, riak, rabbitmq • Monitoring: ganglia, graphite, graphtastic, nagios, opentsdb, statsd, zabbix • Notification: email, hipchat, irc, pagerduty, sns • Protocol: gelf, http, lumberjack, metriccatcher, stomp, tcp, udp, websocket, xmpp • External Monitoring: boundary, circonus, cloudwatch, datadog, librato • External service: google big query, google cloud storage, jira, loggly, riemann, s3, sqs, syslog, zeromq • Local: csv, exec, file, pipe, stdout, null Logstash: Outputs 21
22. www.elastic.co Logstash: It’s Alive (as well)! $ wget https://download.elasticsearch.org/... $ tar -xf logstash-1.4.2.tar.gz $ ./logstash-1.4.2/bin/logstash -f sample.conf 22
23. www.elastic.co Logstash: A Simple Example input { stdin {} } output { stdout { debug => true } } echo foo | logstash-1.4.4/bin/logstash -f sample.conf { "message" => "foo", "@version" => "1", "@timestamp" => "2015-01-10T13:30:59.648Z", "host" => “kryptic.elasticsearch.org” } 23
24. www.elastic.co Logstash: Do You Grok? input { stdin {} } filter { grok { match => [ "message", "%{WORD:firstname} %{WORD:lastname} %{NUMBER:age}" ] } } output { stdout { debug => true } } 24
25. www.elastic.co Logstash: Grok It echo “Nick Fury 100" | logstash-1.4.2/bin/logstash -f sample.conf { "message" => “Nick Fury 100", "@version" => "1", "@timestamp" => "2014-01-10T16:56:02.502Z", "host" => "kryptic", "firstname" => "Nick", "lastname" => "Fury", "age" => "100" } 25
26. www.elastic.co Logstash: Groking Gets Serious input { stdin {} } filter { grok { match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" } } date { match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ] } } output { stdout { debug => true } } Jan 10 04:04:01 lvps109-104-93-171 postfix/smtpd[11105]: connect from mail- we0-f196.google.com[74.125.82.196] 26
27. www.elastic.co Logstash: Added Value cat sample-syslog.txt| logstash-1.4.2/bin/logstash -f sample-syslog.conf { "message" => "Jan 10 04:04:01 lvps109-104-93-171 postfix/smtpd[11105]: connect from mail-we0-f196.google.com[74.125.82.196]", "@version" => "1", "@timestamp" => "2015-01-10T04:04:01.000+02:00", "host" => “kryptic.elasticsearch.org", "syslog_timestamp" => "Jun 10 04:04:01", "syslog_hostname" => "lvps109-104-93-171", "syslog_program" => "postfix/smtpd", "syslog_pid" => "11105", "syslog_message" => "connect from mail-we0-f196.google.com[74.125.82.196]" } 27
28. www.elastic.co Logstash: Apache CLF Parsing { "message" => "193.99.144.85 - - [23/Jan/2014:17:11:55 +0000] \"GET / HTTP/1.1\" 200 140 \"-\" \"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1025.5 Safari/535.19\"", "@version" => "1", "@timestamp" => "2014-01-24T07:56:02.460Z", "host" => "kryptic.local", "clientip" => "193.99.144.85", "ident" => "-", "auth" => "-", "timestamp" => "23/Jan/2014:17:11:55 +0000", "verb" => "GET", "request" => "/", "httpversion" => "1.1", "response" => "200", "bytes" => "140", "referrer" => "\"-\"", "agent" => "\"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1025.5 Safari/535.19\"" } 28
29. www.elastic.co Logstash: Storing in Elasticsearch input { stdin {} } filter { grok { match => [ message, "%{COMBINEDAPACHELOG}" ] } } output { elasticsearch { protocol => “http” } } 29
30. www.elastic.co Elasticsearch: In 30 Seconds • Schema-free, REST & JSON based document store • Distributed and horizontally scalable • Open Source: Apache License 2.0 • Zero configuration • Written in Java, extensible • APIs for everything 30
31. www.elastic.co Elasticsearch: Basic Terms • Index –Logical collection of data; might be time based –Analogous to a database • Shard(s) –Split logical data (index) over several machines –Write scalability • Replica(s) –Read scalability –Removing SPOF 31
32. www.elastic.co Elasticsearch: Cluster Management • Single master at any point in time –Responsible for cluster state (node entry, index creation) • Multicast or unicast based discovery • Configuration is required here –Multicast - Tell each node the name of the cluster to join –Unicast - use IP(s) of existing nodes to join • Tip: Keep master-eligible node count uneven, helps to prevent split brain 32
33. www.elastic.co Elasticsearch: Sizing a Cluster • Data and operation dependent –How big are your documents? How many fields in them? –What is your query rate? –Do you do facets/aggregations, sorting, custom scoring? –What is your write rate? –Do you delete documents? Update them? –Is the data time-based? • Test on one node, one shard, no replicas –Look at shard size, JVM heap usage and GC frequency, number of shards/node, docs per shard, CPU and disk utilization • Tip: No more than 31 GB JVM heap 33
34. www.elastic.co Elasticsearch: Ecosystem • Plugins –Many third party plugins available –Languages, monitoring, attachments, transport, scripting –Build your own! • Clients for many languages –Ruby, python, php, perl, javascript –Scala, clojure, go, .NET • Hadoop integration –Elasticsearch for Apache Hadoop 34
35. www.elastic.co Elasticsearch: Installation $ wget https://download.elasticsearch.org/... $ tar -xf elasticsearch-1.5.0.tar.gz $ ./elasticsearch-1.5.0/bin/elasticsearch ... [2015-03-31 14:53:11,508][INFO ][node] [Scanner] started ... 2 minutes to live! 35
36. www.elastic.co Elasticsearch: It's Alive! » curl localhost:9200 { "status" : 200, "name" : "Scanner", "version" : { "number" : “1.5.0", "build_hash" : "544816042d40151d3ce4ba4f95399d7860dc2e92", "build_timestamp" : "2015-03-23T14:30:58Z", "build_snapshot" : false, "lucene_version" : “4.10.4" }, "tagline" : "You Know, for Search" } 36
37. www.elastic.co Elasticsearch: REST-based Management • Elasticsearch is full of monitoring APIs –Everything is returned as JSON • Humans are not the world’s best JSON parsers • TIP: use ?pretty on end of curl requests 37
38. www.elastic.co Elasticsearch: The _cat API • /_cat/aliases • /_cat/allocation • /_cat/count • /_cat/fielddata • /_cat/health • /_cat/indices • /_cat/master • /_cat/nodes • /_cat/pending_tasks • /_cat/plugins • /_cat/recovery • /_cat/shards • /_cat/thread_pool 38
39. www.elastic.co Elasticsearch: Scaling • Provision a new node • Point it to existing node/cluster • Shards will auto balance • Query/insert via any node • Survive node loss with replicas 39
40. www.elastic.co Example ELK Logging Architecture ELK Stack Kibana server Kibana 4 Logstash “cluster” Logstash Logstash 2x Elasticsearch cluster Client node Client node 2x Data node Data node 5x Master node Master node 3x Message brokers RabbitMQ/ Kafka/Redis RabbitMQ/ Kafka/Redis 2x Server Shipper (LSF, nxlog, syslog, Flume, etc) Shipper (LSF, nxlog, syslog, Flume, etc) 4x 40
41. www.elastic.co Kibana • Demo! 41
42. www.elastic.co Thanks! (special thanks to Mark Walkom for letting me steal slides from his awesome deck, Corralling Logs with ELK: https://speakerdeck.com/elastic/corralling-logs-with-elk) 42