A Technical Overview of the Elastic Platform

This is a high level overview of the technical capabilities of Elasticsearch, Logstash and Kibana as well as introductions into Marvel, Shield and Watcher.

Presented at the Adelaide Big Data Meetup - http://www.meetup.com/Big-Data-Adelaide/events/224255349/

Transcript

1. Elasticsearch, Logstash, Kibana
 Technical Walk-Through Mark Walkom, Hat Wearer @warkolm
2. www.elastic.co 2 Elasticsearch
3. www.elastic.co 3 Elasticsearch Terminology •A node is a single Elasticsearch instance, a single JVM •Multiple nodes can form a cluster •A cluster can manage multiple indices •A cluster is agile & self managing •Clusters often 3-10 nodes but can scale to 100s of nodes •Clusters can have Petabytes of data •Clusters can be federated for larger scale
4. www.elastic.co 4 an open source, distributed, scalable, highly available, document-oriented, RESTful full text search engine with real-time search and analytics capabilities built on lucene and java Elasticsearch is...
5. www.elastic.co 5 an open source, distributed, scalable, highly available, document-oriented, RESTful, full text search engine with real-time search and analytics capabilities Elasticsearch is... Apache 2.0 License   https://www.apache.org/licenses/LICENSE-2.0
6. www.elastic.co 6 an open source, distributed, scalable, highly available, document-oriented, RESTful, full text search engine with real-time search and analytics capabilities Elasticsearch is...
7. www.elastic.co 7 an open source, distributed, scalable, highly available, document-oriented, RESTful, full text search engine with real-time search and analytics capabilities Elasticsearch is...
8. www.elastic.co 8 an open source, distributed, scalable, highly available, document-oriented, RESTful, full text search engine with real-time search and analytics capabilities Elasticsearch is...
9. www.elastic.co 9 an open source, distributed, scalable, highly available, document-oriented, RESTful, full text search engine with real-time search and analytics capabilities Elasticsearch is... Source:  http://json.org/
10. www.elastic.co 10 an open source, distributed, scalable, highly available, document-oriented, RESTful, full text search engine with real-time search and analytics capabilities Elasticsearch is... Source:  https://httpwg.github.io/asset/http.svg
11. www.elastic.co 11 an open source, distributed, scalable, highly available, document-oriented, RESTful, full text search engine with real-time search and analytics capabilities Elasticsearch is...
12. www.elastic.co 12 an open source, distributed, scalable, highly available, document-oriented, RESTful, full text search engine with real-time search and analytics capabilities Elasticsearch is...
13. www.elastic.co 13 Search Search with Elasticsearch
14. www.elastic.co 14 CRUD
15. www.elastic.co 15 CRUD
16. www.elastic.co 16 CRUD
17. www.elastic.co 17 CRUD
18. www.elastic.co 18 Searching
19. www.elastic.co 19 Searching
20. www.elastic.co 20 Aggregation Analytics with Elasticsearch
21. www.elastic.co 21 Aggregations GET /person/person/_search?search_type=count
 {   "aggs": {   "by_country": {   "terms": {   "field": "address.country"   }   }   }   } { ..., "aggregations" : {   "by_country" : {   "buckets" : [ {   "key" : "England",   "doc_count" : 30051   }, {   "key" : "Germany",   "doc_count" : 30004   }, {   "key" : "France",   "doc_count" : 15034   }, {   "key" : "Spain",   "doc_count" : 14912   } ]}}} 17% 17% 33% 33% England Germany France Spain
22. www.elastic.co 22 Histograms GET /person/person/_search?search_type=count
 {   "aggs": {   "by_date": {   "date_histogram": {   "field": "dateOfBirth",   "interval": "year",   "format": "yyyy"   }   }   }   } { ..., "aggregations": {   "by_date": {   "buckets": [   {   "key_as_string": "1960",   "key": -946080000000,   "doc_count": 39   },   {   "key_as_string": "1961",   "key": -630720000000,   "doc_count": 12677   },   {   "key_as_string": "1962",   "key": -315360000000,   "doc_count": 12936   }, ...   ]   }   }} 0 7500 15000 22500 30000 1940 1950 1960 1970 1980 1990 2000 2010
23. www.elastic.co 23 A Lot More
24. www.elastic.co 24 More than search Elasticsearch
25. www.elastic.co 25 Text Analysis - Analyzers • Tokenizer Breaks the text into tokens and produces a token stream Example: keyword, whitespace, regex, etc... • Token Filter Acts on the token stream - can drop and modify existing tokens, or add new ones. Example: lowercase, stopword, ngram, etc..
26. www.elastic.co 26 Free steak knives! • Relational documents Parent/child Nesting • Suggestion API Predictive typing/search • Highlighting Emphasise results, e.g. <em>w00t</em> • Percolators - search for searches Does this document match this search?
27. www.elastic.co 27 Geo Search • Geo points and shapes Polygon Polygon with holes Multi polygon • Bounding boxes, distance from point, distance in a range • Supports multiple coordinate formats; “location”: { "lat" : 41.12, "lon" : -71.34 } "location" : “41.12,-71.34” "location" : [-71.34, 41.12]
28. www.elastic.co 28 Elasticsearch & Hadoop
29. www.elastic.co 29 Elasticsearch for Apache Hadoop™
30. www.elastic.co 30 Logstash
31. www.elastic.co 31 Logstash Logstash Input Output Filter ? ? collect and split alter and enrich store and visualise
32. www.elastic.co 32 Logstash
33. www.elastic.co 33 Logstash 71.141.244.242 - kurt [18/May/2011:01:48:10 -0700] "GET /admin HTTP/1.1" 301 566 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3"
34. www.elastic.co 34 Logstash 110.136.166.128 - - [16/Feb/2014:09:48:53 -0500] "GET /images/jordan-80.png HTTP/1.1" 200 6146 "http://www.semicomplete.com/projects/xdotool/" "Mozilla/5.0 (Windows NT 6.2; WOW64; rv:28.0) Gecko/20100101 Firefox/28.0" 123.125.71.35 - - [16/Feb/2014:09:49:02 -0500] "GET /blog/tags/release HTTP/1.1" 200 40693 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)" 110.136.166.128 - - [16/Feb/2014:09:48:53 -0500] "GET /images/web/2009/banner.png HTTP/1.1" 200 52315 "http://www.semicomplete.com/style2.css" "Mozilla/5.0 (Windows NT 6.2; WOW64; rv:28.0) Gecko/20100101 Firefox/28.0" 50.150.204.184 - - [16/Feb/2014:09:49:37 -0500] "GET /images/googledotcom.png HTTP/1.1" 200 65748 "http://www.google.com/search?q=https//:google.com&source=lnms&tbm=isch&sa=X&ei=4-r8UvDrKZOgkQe7x4CICw&ved=0CAkQ_AUoAA&biw=320&bih=441" "Mozilla/5.0 (Linux; U; Android 4.0.4; en-us; LG-MS770 Build/IMM76I) AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 Mobile Safari/534.30" 207.241.237.225 - - [16/Feb/2014:09:50:06 -0500] "GET /blog/tags/examples HTTP/1.0" 200 9208 "http://www.semicomplete.com/blog/tags/C" "Mozilla/5.0 (compatible; archive.org_bot +http://www.archive.org/details/archive.org_bot)" 200.49.190.101 - - [16/Feb/2014:09:50:10 -0500] "GET /reset.css HTTP/1.1" 200 1015 "-" "-" 200.49.190.100 - - [16/Feb/2014:09:50:08 -0500] "GET /blog/tags/web HTTP/1.1" 200 44019 "-" "QS304 Profile/MIDP-2.0 Configuration/CLDC-1.1" 200.49.190.101 - - [16/Feb/2014:09:50:12 -0500] "GET /style2.css HTTP/1.1" 200 4877 "-" "-" 200.49.190.101 - - [16/Feb/2014:09:50:19 -0500] "GET /images/jordan-80.png HTTP/1.1" 200 6146 "-" "QS304 Profile/MIDP-2.0 Configuration/CLDC-1.1" 66.249.73.185 - - [16/Feb/2014:09:51:19 -0500] "GET /reset.css HTTP/1.1" 200 1015 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 66.249.73.135 - - [16/Feb/2014:09:51:26 -0500] "GET /blog/tags/munin HTTP/1.1" 200 9746 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 66.249.73.135 - - [16/Feb/2014:09:51:47 -0500] "GET /blog/tags/firefox?flav=rss20 HTTP/1.1" 200 16021 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 66.249.73.135 - - [16/Feb/2014:09:52:34 -0500] "GET /blog/geekery/eventdb-ideas.html HTTP/1.1" 200 11418 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 67.214.178.190 - - [16/Feb/2014:09:53:19 -0500] "GET / HTTP/1.0" 200 37932 "http://www.semicomplete.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:21.0) Gecko/20100101 Firefox/21.0" 67.214.178.190 - - [16/Feb/2014:09:53:30 -0500] "GET /blog/geekery/installing-windows-8-consumer-preview.html HTTP/1.0" 200 8948 "http://www.semicomplete.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:21.0) Gecko/20100101 Firefox/21.0" 207.241.237.220 - - [16/Feb/2014:09:53:47 -0500] "GET /blog/tags/projects HTTP/1.0" 200 28370 "http://www.semicomplete.com/blog/tags/C" "Mozilla/5.0 (compatible; archive.org_bot +http://www.archive.org/details/archive.org_bot)" 46.105.14.53 - - [16/Feb/2014:09:53:48 -0500] "GET /blog/tags/puppet?flav=rss20 HTTP/1.1" 200 14872 "-" "UniversalFeedParser/4.2-pre-314-svn +http://feedparser.org/" 207.241.237.227 - - [16/Feb/2014:09:53:50 -0500] "GET /blog/geekery/soekris-gpio.html HTTP/1.0" 200 9587 "http://www.semicomplete.com/blog/tags/C" "Mozilla/5.0 (compatible; archive.org_bot +http://www.archive.org/details/archive.org_bot)" 91.177.205.119 - - [16/Feb/2014:09:54:34 -0500] "GET /blog/geekery/xvfb-firefox.html HTTP/1.1" 200 10975 "http://en.wikipedia.org/wiki/Xvfb" "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0)" 91.177.205.119 - - [16/Feb/2014:09:54:34 -0500] "GET /reset.css HTTP/1.1" 200 1015 "http://semicomplete.com/blog/geekery/xvfb-firefox.html" "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0)" 91.177.205.119 - - [16/Feb/2014:09:54:34 -0500] "GET /style2.css HTTP/1.1" 200 4877 "http://semicomplete.com/blog/geekery/xvfb-firefox.html" "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0)" 91.177.205.119 - - [16/Feb/2014:09:54:34 -0500] "GET /images/jordan-80.png HTTP/1.1" 200 6146 "http://semicomplete.com/blog/geekery/xvfb-firefox.html" "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0)" 91.177.205.119 - - [16/Feb/2014:09:54:34 -0500] "GET /images/web/2009/banner.png HTTP/1.1" 200 52315 "http://semicomplete.com/blog/geekery/xvfb-firefox.html" "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0)" 91.177.205.119 - - [16/Feb/2014:09:54:35 -0500] "GET /favicon.ico HTTP/1.1" 200 3638 "-" "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Win64; x64; Trident/6.0)" 66.249.73.185 - - [16/Feb/2014:09:54:44 -0500] "GET /doc/index.html?org/elasticsearch/action/search/SearchResponse.html HTTP/1.1" 404 294 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 207.241.237.228 - - [16/Feb/2014:09:54:54 -0500] "GET /blog/tags/defcon HTTP/1.0" 200 24142 "http://www.semicomplete.com/blog/tags/C" "Mozilla/5.0 (compatible; archive.org_bot +http://www.archive.org/details/archive.org_bot)" 207.241.237.101 - - [16/Feb/2014:09:54:58 -0500] "GET /blog/tags/regex HTTP/1.0" 200 14888 "http://www.semicomplete.com/blog/tags/C" "Mozilla/5.0 (compatible; archive.org_bot +http://www.archive.org/details/archive.org_bot)" 87.169.99.232 - - [16/Feb/2014:09:56:12 -0500] "GET /presentations/puppet-at-loggly/puppet-at-loggly.pdf.html HTTP/1.1" 200 24747 "https://www.google.de/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36" 209.85.238.199 - - [16/Feb/2014:09:56:18 -0500] "GET /blog/tags/firefox?flav=rss20 HTTP/1.1" 200 16021 "-" "Feedfetcher-Google; (+http://www.google.com/feedfetcher.html; 3 subscribers; feed-id=14171215010336145331)" 209.85.238.199 - - [16/Feb/2014:09:56:31 -0500] "GET /test.xml HTTP/1.1" 200 1370 "-" "Feedfetcher-Google; (+http://www.google.com/feedfetcher.html; 1 subscribers; feed-id=11390274670024826467)" 81.220.24.207 - - [16/Feb/2014:09:57:28 -0500] "GET /blog/geekery/ssl-latency.html HTTP/1.1" 200 17147 "http://www.google.fr/url?sa=t&rct=j&q=&esrc=s&source=web&cd=5&ved=0CE4QFjAE&url=http%3A%2F%2Fwww.semicomplete.com%2Fblog%2Fgeekery%2Fssl-latency.html&ei=ZdEAU9mGGuWX1AW09IDoBw&usg=AFQjCNHw6zioJpizqX8Q0YpKKaF4zdCSEg&bvm=bv.61535280,d.d2k" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/ 537.73.11 (KHTML, like Gecko) Version/7.0.1 Safari/537.73.11" 81.220.24.207 - - [16/Feb/2014:09:57:28 -0500] "GET /reset.css HTTP/1.1" 200 1015 "http://www.semicomplete.com/blog/geekery/ssl-latency.html" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.73.11 (KHTML, like Gecko) Version/7.0.1 Safari/537.73.11" 81.220.24.207 - - [16/Feb/2014:09:57:28 -0500] "GET /images/jordan-80.png HTTP/1.1" 200 6146 "http://www.semicomplete.com/blog/geekery/ssl-latency.html" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.73.11 (KHTML, like Gecko) Version/7.0.1 Safari/537.73.11" 81.220.24.207 - - [16/Feb/2014:09:57:28 -0500] "GET /style2.css HTTP/1.1" 200 4877 "http://www.semicomplete.com/blog/geekery/ssl-latency.html" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.73.11 (KHTML, like Gecko) Version/7.0.1 Safari/537.73.11" 81.220.24.207 - - [16/Feb/2014:09:57:28 -0500] "GET /images/web/2009/banner.png HTTP/1.1" 200 52315 "http://www.semicomplete.com/blog/geekery/ssl-latency.html" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.73.11 (KHTML, like Gecko) Version/7.0.1 Safari/537.73.11" 81.220.24.207 - - [16/Feb/2014:09:57:29 -0500] "GET /favicon.ico HTTP/1.1" 200 3638 "http://www.semicomplete.com/blog/geekery/ssl-latency.html" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.73.11 (KHTML, like Gecko) Version/7.0.1 Safari/537.73.11" 66.249.73.135 - - [16/Feb/2014:09:57:36 -0500] "GET /blog/geekery/vmware-cpu-performance.html HTTP/1.1" 200 12908 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 46.105.14.53 - - [16/Feb/2014:09:58:48 -0500] "GET /blog/tags/puppet?flav=rss20 HTTP/1.1" 200 14872 "-" "UniversalFeedParser/4.2-pre-314-svn +http://feedparser.org/" 218.30.103.62 - - [16/Feb/2014:09:59:36 -0500] "GET /robots.txt HTTP/1.1" 200 - "-" "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)" 218.30.103.62 - - [16/Feb/2014:09:59:41 -0500] "GET /robots.txt HTTP/1.1" 200 - "-" "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)" 218.30.103.62 - - [16/Feb/2014:09:59:46 -0500] "GET /projects/fex/ HTTP/1.1" 200 14352 "-" "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)" 74.125.40.20 - - [16/Feb/2014:09:59:53 -0500] "GET /?flav=rss20 HTTP/1.1" 200 29941 "-" "FeedBurner/1.0 (http://www.FeedBurner.com)" 71.212.224.97 - - [16/Feb/2014:10:00:05 -0500] "GET /projects/xdotool/ HTTP/1.1" 200 12292 "http://suckless.org/rocks" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36" 71.212.224.97 - - [16/Feb/2014:10:00:05 -0500] "GET /reset.css HTTP/1.1" 200 1015 "http://www.semicomplete.com/projects/xdotool/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36" 71.212.224.97 - - [16/Feb/2014:10:00:06 -0500] "GET /style2.css HTTP/1.1" 200 4877 "http://www.semicomplete.com/projects/xdotool/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36" 71.212.224.97 - - [16/Feb/2014:10:00:06 -0500] "GET /images/jordan-80.png HTTP/1.1" 200 6146 "http://www.semicomplete.com/projects/xdotool/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36" 71.212.224.97 - - [16/Feb/2014:10:00:06 -0500] "GET /images/web/2009/banner.png HTTP/1.1" 200 52315 "http://www.semicomplete.com/projects/xdotool/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36" 218.30.103.62 - - [16/Feb/2014:10:00:07 -0500] "GET /projects/xdotool/xdotool.xhtml HTTP/1.1" 304 - "-" "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)" 108.174.55.234 - - [16/Feb/2014:10:00:16 -0500] "GET /?flav=rss20 HTTP/1.1" 200 29941 "-" "-" 218.30.103.62 - - [16/Feb/2014:10:00:28 -0500] "GET /blog/geekery/c-vs-python-bdb.html HTTP/1.1" 200 11388 "-" "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)" 121.107.188.202 - - [16/Feb/2014:10:00:28 -0500] "GET /presentations/logstash-monitorama-2013/images/kibana-dashboard3.png HTTP/1.1" 200 171717 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36" 218.30.103.62 - - [16/Feb/2014:10:00:52 -0500] "GET /blog/productivity/better-zsh-xterm-title-fix.html HTTP/1.1" 200 10185 "-" "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)" 218.30.103.62 - - [16/Feb/2014:10:01:14 -0500] "GET /blog/geekery/xvfb-firefox.html HTTP/1.1" 200 10975 "-" "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)" 218.30.103.62 - - [16/Feb/2014:10:01:37 -0500] "GET /blog/geekery/puppet-facts-into-mcollective.html HTTP/1.1" 200 9872 "-" "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)" 198.46.149.143 - - [16/Feb/2014:10:01:44 -0500] "GET /blog/geekery/disabling-battery-in-ubuntu-vms.html?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+semicomplete%2Fmain+%28semicomplete.com+-+Jordan+Sissel%29 HTTP/1.1" 200 9316 "-" "Tiny Tiny RSS/1.11 (http://tt-rss.org/)" 198.46.149.143 - - [16/Feb/2014:10:01:44 -0500] "GET /blog/geekery/solving-good-or-bad-problems.html?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+semicomplete%2Fmain+%28semicomplete.com+-+Jordan+Sissel%29 HTTP/1.1" 200 10756 "-" "Tiny Tiny RSS/1.11 (http://tt-rss.org/)" 218.30.103.62 - - [16/Feb/2014:10:01:57 -0500] "GET /blog/geekery/jquery-interface-puffer.html%20target= HTTP/1.1" 200 202 "-" "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)" 218.30.103.62 - - [16/Feb/2014:10:02:19 -0500] "GET /blog/geekery/ec2-reserved-vs-ondemand.html HTTP/1.1" 200 11834 "-" "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)" 66.249.73.135 - - [16/Feb/2014:10:02:37 -0500] "GET /blog/web/firefox-scrolling-fix.html HTTP/1.1" 200 8956 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 86.1.76.62 - - [16/Feb/2014:10:03:08 -0500] "GET /projects/xdotool/ HTTP/1.1" 200 12292 "http://www.haskell.org/haskellwiki/Xmonad/Frequently_asked_questions" "Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20140205 Firefox/24.0 Iceweasel/24.3.0" 86.1.76.62 - - [16/Feb/2014:10:03:08 -0500] "GET /reset.css HTTP/1.1" 200 1015 "http://www.semicomplete.com/projects/xdotool/" "Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20140205 Firefox/24.0 Iceweasel/24.3.0" 86.1.76.62 - - [16/Feb/2014:10:03:08 -0500] "GET /style2.css HTTP/1.1" 200 4877 "http://www.semicomplete.com/projects/xdotool/" "Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20140205 Firefox/24.0 Iceweasel/24.3.0" 86.1.76.62 - - [16/Feb/2014:10:03:08 -0500] "GET /favicon.ico HTTP/1.1" 200 3638 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20140205 Firefox/24.0 Iceweasel/24.3.0" 86.1.76.62 - - [16/Feb/2014:10:03:08 -0500] "GET /images/jordan-80.png HTTP/1.1" 200 6146 "http://www.semicomplete.com/projects/xdotool/" "Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20140205 Firefox/24.0 Iceweasel/24.3.0" 86.1.76.62 - - [16/Feb/2014:10:03:08 -0500] "GET /images/web/2009/banner.png HTTP/1.1" 200 52315 "http://www.semicomplete.com/style2.css" "Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20140205 Firefox/24.0 Iceweasel/24.3.0” 66.249.73.135 - - [16/Feb/2014:10:03:25 -0500] "GET /blog/tags/bdb HTTP/1.1" 200 23099 "-" "DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)" 107.170.41.69 - - [16/Feb/2014:10:03:31 -0500] "GET /?flav=atom HTTP/1.1" 200 32352 "-" "Feedbin - 1 subscribers" 50.16.19.13 - - [16/Feb/2014:10:03:43 -0500] "GET /blog/tags/puppet?flav=rss20 HTTP/1.1" 200 14872 "http://www.semicomplete.com/blog/tags/puppet?flav=rss20" "Tiny Tiny RSS/1.11 (http://tt-rss.org/)" 46.105.14.53 - - [16/Feb/2014:10:03:50 -0500] "GET /blog/tags/puppet?flav=rss20 HTTP/1.1" 200 14872 "-" "UniversalFeedParser/4.2-pre-314-svn +http://feedparser.org/"
35. www.elastic.co 35 Logstash input { stdin { } } filter { grok { match => { "message" => '%{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] "% {WORD:verb} %{DATA:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:response:int} (?:-|% {NUMBER:bytes:int}) %{QS:referrer} %{QS:agent}' } } date { match => [ "timestamp", "dd/MMM/YYYY:HH:mm:ss Z" ] locale => en } geoip { source => "clientip" } useragent { source => "agent" target => "useragent" } } output { stdout { codec => rubydebug } }
36. www.elastic.co 36 Logstash { "message" => "71.141.244.242 - kurt [18/May/2011:01:48:10 -0700] \"GET /admin HTTP/1.1\" 301 566 \"-\" \"Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3\"", "@version" => "1", "@timestamp" => "2011-05-18T08:48:10.000Z", "host" => "bender.local", "clientip" => "71.141.244.242", "ident" => "-", "auth" => "kurt", "timestamp" => "18/May/2011:01:48:10 -0700", "verb" => "GET", "request" => "/admin", "httpversion" => "1.1", "response" => 301, "bytes" => 566, "referrer" => "\"-\"", "agent" => "\"Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3\"", "geoip" => { "ip" => "71.141.244.242", "country_code2" => “US", "city_name" => "San Francisco", "timezone" => "America/Los_Angeles", "location" => [ [0] -122.4194, [1] 37.7749 ] }, "useragent" => { "name" => "Firefox", "os" => "Windows XP", "os_name" => "Windows XP", "device" => "Other", "major" => "3", "minor" => "6", "patch" => "3" } }
37. www.elastic.co 37 Logstash input { stdin {} } filter { grok { match => [ message, "%{COMBINEDAPACHELOG}" ] } } output { elasticsearch { protocol => “http” host => “bender” } }
38. www.elastic.co 38 Kibana
39. www.elastic.co 39 Kibana •Kibana 4 is a total re-architecture from 3 Nodejs + javascript Zazzier UI Single binary that serves itself •Lots more functionality via aggregations •Extensible - plugins coming real soon
40. www.elastic.co 40 Kibana
41. www.elastic.co 41 Kibana
42. www.elastic.co 42 Kibana
43. www.elastic.co 43 Found - ESaaS •Fully Managed and Monitored Infrastructure Automated Backups HA - Replication and Failover •GUI Driven, User Friendly* •Sydney AZ very, very soon
44. www.elastic.co 44 Elastic: Commercial Plugins •Marvel: Monitor your Cluster Currently KB3 based front end. v2.0 will be KB4. •Shield: For Security ACLs, RBAC via AD or LDAP, SSL, IP filtering, Auditing •Watcher: Alerting on your data Email and webhook push notifications •More coming soon!
45. www.elastic.co 45 Goodies •Curator: index management https://www.elastic.co/guide/en/elasticsearch/client/ curator/current/index.html •Puppet & Chef modules https://forge.puppetlabs.com/elasticsearch https://github.com/elastic/cookbook-elasticsearch/ •logstash forwarder: low overhead collector https://github.com/elastic/logstash-forwarder •grokdebugger: log pattern matching http://grokdebug.herokuapp.com/
46. www.elastic.co 46 More Goodies •Github: https://github.com/elastic •Docs: http://www.elastic.co/guide/ •Forums: https://discuss.elastic.co •IRC channels #elasticsearch, #logstash, #kibana, #beats on Freenode •We’re hiring! jobs@elastic.co, drop me an email/DM or come say Hi :)
47. Thanks! Mark Walkom, Hat Wearer @warkolm