The ELK Stack: For Real-Time Enlightenment

The slides from my talk presenting the ELK stack a the DevOps Ireland meetup: http://www.meetup.com/DevOps-Ireland/events/201680842/

Transcript

1. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Boaz Leskes @bleskes boaz.leskes@elasticsearch.com The ELK Stack for Real Time Enlightenment
2. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Elasticsearch in 10 seconds • Schema-free, REST & JSON based document store • Distributed and horizontally scalable • Open Source: Apache License 2.0 • Zero configuration • Written in Java, extensible
3. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited. • full text search • highlighted search snippets • search-as-you-type • did-you-mean suggestions
4. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited. • combines full text search with geolocation • uses more-like-this to find 
 related questions and answers
5. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited. • search repositories, users, 
 issues, pull requests • search 130 billion lines of code • track all alerts, events, logs
6. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited. • index and analyse 
 5TB of log data every day
7. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited. • combine visitor logs with 
 social network data • real-time feedback to editors
8. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
9. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Feature summary • Fully-featured search Relevance-ranked text search Scalable search High-performance geo, temporal, numeric range and key lookup Highlighting Support for complex document types (nested structures) * Spelling suggestions Powerful query DSL * “Standing” queries * Real-time results * Extensible via plugins * ! • Powerful faceting/analysis Summarise large sets by any combinations of time, geo, category and more. * “Kibana” visualisation tool * ! • Management Simple and robust deployments * REST APIs for handling all aspects of administration/ monitoring * “Marvel” console for monitoring and administering clusters * Special features to manage the life cycle of content * • Integration Hadoop (MapRed,Hive, Pig, Cascading..)* Client libraries (Python, Java, Ruby, javascript…) Data connectors (Twitter, JMS…) Logstash ETL framework * • Support Development and Production support with tiered levels Support staff are the core developers of the product * * Features we see as differentiators
10. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Unstructured search
11. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Aggregation to find languages
12. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Structured search
13. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Enrichment
14. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Sorting
15. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Pagination
16. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Suggestions
17. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited … or more complex …
18. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Installation & first steps
19. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited 2 minutes to live $ wget https://download.elasticsearch.org/... $ tar -xf elasticsearch-1.3.2.tar.gz $ ./elasticsearch-1.3.2/bin/elasticsearch ... [2014-04-16 14:53:11,508][INFO ][node] [Scanner] started ... Also puppet modules and RPM/DEB
20. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Is it alive? » curl localhost:9200 { "status" : 200, "name" : "Scanner", "version" : { "number" : “1.3.2", "build_hash" : "dee175dbe2f254f3f26992f5d7591939aaefd12f", "build_timestamp" : "2014-08-13T14:29:30Z", "build_snapshot" : true, "lucene_version" : “4.9" }, "tagline" : "You Know, for Search" }
21. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited » curl -XPUT localhost:9200/books/book/1 -d ' { "title" : "Elasticsearch - The definitive guide", "authors" : "Clinton Gormley", "started" : "2013-02-04", "pages" : 230 }' Create…
22. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited » curl -XPUT localhost:9200/books/book/1 -d ' { "title" : "Elasticsearch - The definitive guide", "authors" : [ "Clinton Gormley", "Zachary Tong" ], "started" : "2013-02-04", "pages" : 230 }' Update…
23. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Delete… » curl -X DELETE localhost:9200/books/book/1 Realtime GET… » curl —X GET localhost:9200/books/book/1 » curl —X GET localhost:9200/books/book/1/_source
24. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Search » curl -XGET localhost:9200/books/_search?q=elasticsearch { "took" : 2, "timed_out" : false, "_shards" : { "total" : 5, "successful" : 5, "failed" : 0 }, "hits" : { "total" : 1, "max_score" : 0.076713204, "hits" : [ { "_index" : “books", "_type" : “book", "_id" : "1", "_score" : 0.076713204, "_source" : { "title" : "Elasticsearch - The definitive guide", "authors" : [ "Clinton Gormley", "Zachary Tong" ], "started" : “2013-02-04", "pages" : 230 } } ] } }
25. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited » curl -XGET ‘localhost:9200/books/book/_search' -d '{ "query": { "filtered" : { "query" : { "match": { "text" : { "query" : “To Be Or Not To Be", "cutoff_frequency" : 0.01 } } }, "filter" : { "range": { "price": { "gte": 20.0 "lte": 50.0 ... } }' Search - Query DSL
26. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Distributed and scalable
27. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Basic terms • Index Logical collection of data; might be time based Analogous to a database • Sharding Split logical data over several machines Write scalability Control data flows • Replication Read scalability Removing SPOF
28. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Shards and replicas node 1 orders products 1 4 1 2 2 2 curl -X PUT localhost:9200/orders -d '{ "settings.index.number_of_shards" : 4 "settings.index.number_of_replicas" : 1 }' curl -X PUT localhost:9200/products -d '{ "settings.index.number_of_shards" : 2 "settings.index.number_of_replicas" : 0 }'
29. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Shards and replicas node 1 orders products 1 4 1 node 2 orders products 2 2 3 4 1 2 3
30. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Automatic leveling node 1 orders products 2 1 4 1 node 2 orders products 2 2 node 3 orders products 3 4 1 3
31. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited What is data? • Whatever provides value for your business ! • Domain data Internal: Orders, products External: Social media streams, email • Application data Log files Metrics
32. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Ecosystem • Plugins Many third party plugins available • Clients for many languages Ruby, python, php, perl, javascript, (.NET coming) Scala, clojure, go • Kibana • Logstash • Hadoop integration
33. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Monitor your cluster with Marvel • Point in time views are a start • Marvel shows historical trends • Visualize cluster behavior, act before problems ! • Free for development, $500/year for up to 5 nodes
34. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Overview
35. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Node statistics
36. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Sense
37. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Log analysis with
    Logstash and Kibana
38. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Logstash in 10 seconds • Managing events and logs • Collect, parse, enrich, store data • Modular: many, many inputs and outputs • Apache License 2.0 • Ruby app (JRuby) • Part of Elasticsearch family
39. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited What is a log? • Time-based data • This data is everywhere! Server logs Twitter stream Financial transactions Metric / monitoring data ... • Log all things
40. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Why collect & centralize logs? • Access log files without system access • Shell scripting: Too limited or slow • Using unique ids for errors, aggregate it across your stack • Reporting (everyone can create his/her own report) • Tip: Unify your data to make it easily searchable
41. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Logstash architecture Logstash Input Output Filter ? ? collect and split alter and enrich store and visualize
42. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Inputs • Monitoring: collectd, graphite, ganglia, snmptrap, zenoss • Datastores: elasticsearch, redis, sqlite, s3 • Queues: rabbitmq, zeromq • Logging: eventlog, lumberjack, gelf, log4j, relp, syslog, varnish log • Platforms: drupal_dblog, gemfire, heroku, sqs, s3, twitter • Local: exec, generator, file, stdin, pipe, unix • Protocol: imap, irc, stomp, tcp, udp, websocket, wmi, xmpp
43. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Filters • alter, anonymize, checksum, csv, drop, multiline • dns, date, extractnumbers, geoip, i18n, kv, noop, ruby, range • json, urldecode, useragent • metrics, sleep • … many, many more …
44. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Outputs • Store: elasticsearch, gemfire, mongodb, redis, riak, rabbitmq • Monitoring: ganglia, graphite, graphtastic, nagios, opentsdb, statsd, zabbix • Notification: email, hipchat, irc, pagerduty, sns • Protocol: gelf, http, lumberjack, metriccatcher, stomp, tcp, udp, websocket, xmpp • External Monitoring: boundary, circonus, cloudwatch, datadog, librato • External service: google big query, google cloud storage, jira, loggly, riemann, s3, sqs, syslog, zeromq • Local: csv, exec, file, pipe, stdout, null
45. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Demo
46. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Visualize with Kibana
47. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Kibana in 10 seconds • Visualize data in Elasticsearch • See real-time updates to the data • Build custom charts and dashboards • Apache License 2.0 • Runs in browser (Chrome, FF, IE, Safari) • Part of Elasticsearch family
48. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Useful helpers • Curator: index management http://www.elasticsearch.org/blog/curator-tending-your-time-series-indices/ • Puppet module https://github.com/elasticsearch/puppet-logstash • Logstash forwarder: low overhead collector https://github.com/elasticsearch/logstash-forwarder • Logstash cookbook http://cookbook.logstash.net/
49. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited More info • Github: https://github.com/elasticsearch Code, issues there • Mailing lists Google groups, logstash-users and elasticsearch • IRC channels #logstash and #elasticsearch on freenode • We’re hiring! jobs@elasticsearch.com