The ELK Stack: For Real-Time Enlightenment

The slides from my talk presenting the ELK stack a the DevOps Ireland meetup: http://www.meetup.com/DevOps-Ireland/events/201680842/

Transcript

1. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
   Boaz Leskes
   @bleskes
   boaz.leskes@elasticsearch.com
   The ELK Stack
   for Real Time Enlightenment
   

   View Slide

2. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
   Elasticsearch in 10 seconds
   • Schema-free, REST & JSON based document store
   • Distributed and horizontally scalable
   • Open Source: Apache License 2.0
   • Zero configuration
   • Written in Java, extensible
   

   View Slide

3. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
   • full text search
   • highlighted search snippets
   • search-as-you-type
   • did-you-mean suggestions
   

   View Slide

4. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
   • combines full text search with geolocation
   • uses more-like-this to find 

   related questions and answers
   

   View Slide

5. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
   • search repositories, users, 

   issues, pull requests
   • search 130 billion lines of code
   • track all alerts, events, logs
   

   View Slide

6. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
   • index and analyse 

   5TB of log data every day
   

   View Slide

7. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
   • combine visitor logs with 

   social network data
   • real-time feedback to editors
   

   View Slide

8. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
   

   View Slide

9. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
   Feature summary
   • Fully-featured search
   Relevance-ranked text search
   Scalable search
   High-performance geo, temporal, numeric range and
   key lookup
   Highlighting
   Support for complex document types (nested
   structures) *
   Spelling suggestions
   Powerful query DSL *
   “Standing” queries *
   Real-time results *
   Extensible via plugins *
   !
   • Powerful faceting/analysis
   Summarise large sets by any combinations of time,
   geo, category and more. *
   “Kibana” visualisation tool *
   !
   • Management
   Simple and robust deployments *
   REST APIs for handling all aspects of administration/
   monitoring *
   “Marvel” console for monitoring and administering
   clusters *
   Special features to manage the life cycle of content *
   • Integration
   Hadoop (MapRed,Hive, Pig, Cascading..)*
   Client libraries (Python, Java, Ruby, javascript…)
   Data connectors (Twitter, JMS…)
   Logstash ETL framework *
   • Support
   Development and Production support with tiered levels
   Support staff are the core developers of the product *
   * Features we see as differentiators
   

   View Slide

10. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    Unstructured search
    

    View Slide

11. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    Aggregation to find languages
    

    View Slide

12. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    Structured search
    

    View Slide

13. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    Enrichment
    

    View Slide

14. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    Sorting
    

    View Slide

15. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    Pagination
    

    View Slide

16. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    Suggestions
    

    View Slide

17. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    … or more complex …
    

    View Slide

18. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Installation & first steps
    

    View Slide

19. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    2 minutes to live
    $ wget https://download.elasticsearch.org/...
    $ tar -xf elasticsearch-1.3.2.tar.gz
    $ ./elasticsearch-1.3.2/bin/elasticsearch
    ...
    [2014-04-16 14:53:11,508][INFO ][node] [Scanner] started
    ...
    Also puppet modules and RPM/DEB
    

    View Slide

20. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Is it alive?
    » curl localhost:9200
    {
    "status" : 200,
    "name" : "Scanner",
    "version" : {
    "number" : “1.3.2",
    "build_hash" : "dee175dbe2f254f3f26992f5d7591939aaefd12f",
    "build_timestamp" : "2014-08-13T14:29:30Z",
    "build_snapshot" : true,
    "lucene_version" : “4.9"
    },
    "tagline" : "You Know, for Search"
    }
    

    View Slide

21. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    » curl -XPUT localhost:9200/books/book/1 -d '
    {
    "title" : "Elasticsearch - The definitive guide",
    "authors" : "Clinton Gormley",
    "started" : "2013-02-04",
    "pages" : 230
    }'
    Create…
    

    View Slide

22. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    » curl -XPUT localhost:9200/books/book/1 -d '
    {
    "title" : "Elasticsearch - The definitive guide",
    "authors" : [ "Clinton Gormley", "Zachary Tong" ],
    "started" : "2013-02-04",
    "pages" : 230
    }'
    Update…
    

    View Slide

23. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Delete…
    » curl -X DELETE localhost:9200/books/book/1
    Realtime GET…
    » curl —X GET localhost:9200/books/book/1
    » curl —X GET localhost:9200/books/book/1/_source
    

    View Slide

24. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Search
    » curl -XGET localhost:9200/books/_search?q=elasticsearch
    {
    "took" : 2, "timed_out" : false,
    "_shards" : { "total" : 5, "successful" : 5, "failed" : 0 },
    "hits" : {
    "total" : 1, "max_score" : 0.076713204,
    "hits" : [ {
    "_index" : “books", "_type" : “book", "_id" : "1",
    "_score" : 0.076713204, "_source" : {
    "title" : "Elasticsearch - The definitive guide",
    "authors" : [ "Clinton Gormley", "Zachary Tong" ],
    "started" : “2013-02-04", "pages" : 230
    }
    } ]
    }
    }
    

    View Slide

25. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    » curl -XGET ‘localhost:9200/books/book/_search' -d '{
    "query": {
    "filtered" : {
    "query" : {
    "match": {
    "text" : {
    "query" : “To Be Or Not To Be",
    "cutoff_frequency" : 0.01
    }
    }
    },
    "filter" : {
    "range": {
    "price": {
    "gte": 20.0
    "lte": 50.0
    ...
    }
    }'
    Search - Query DSL
    

    View Slide

26. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Distributed and scalable
    

    View Slide

27. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Basic terms
    • Index
    Logical collection of data; might be time based
    Analogous to a database
    • Sharding
    Split logical data over several machines
    Write scalability
    Control data flows
    • Replication
    Read scalability
    Removing SPOF
    

    View Slide

28. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Shards and replicas
    node 1
    orders
    products
    1
    4
    1 2
    2
    2
    curl -X PUT localhost:9200/orders -d '{
    "settings.index.number_of_shards" : 4
    "settings.index.number_of_replicas" : 1
    }'
    curl -X PUT localhost:9200/products -d '{
    "settings.index.number_of_shards" : 2
    "settings.index.number_of_replicas" : 0
    }'
    

    View Slide

29. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Shards and replicas
    node 1
    orders
    products
    1
    4
    1
    node 2
    orders
    products
    2
    2
    3 4
    1 2
    3
    

    View Slide

30. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Automatic leveling
    node 1
    orders
    products
    2
    1
    4
    1
    node 2
    orders
    products
    2
    2
    node 3
    orders
    products
    3 4
    1
    3
    

    View Slide

31. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
    What is data?
    • Whatever provides value for your business
    !
    • Domain data
    Internal: Orders, products
    External: Social media streams, email
    • Application data
    Log files
    Metrics
    

    View Slide

32. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Ecosystem
    • Plugins
    Many third party plugins available
    • Clients for many languages
    Ruby, python, php, perl, javascript, (.NET coming)
    Scala, clojure, go
    • Kibana
    • Logstash
    • Hadoop integration
    

    View Slide

33. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Monitor your cluster with Marvel
    • Point in time views are a start
    • Marvel shows historical trends
    • Visualize cluster behavior, act before problems
    !
    • Free for development, $500/year for up to 5
    nodes
    

    View Slide

34. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Overview
    

    View Slide

35. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Node statistics
    

    View Slide

36. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Sense
    

    View Slide

37. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Log analysis with
    Logstash and Kibana
    

    View Slide

38. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Logstash in 10 seconds
    • Managing events and logs
    • Collect, parse, enrich, store data
    • Modular: many, many inputs and outputs
    • Apache License 2.0
    • Ruby app (JRuby)
    • Part of Elasticsearch family
    

    View Slide

39. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    What is a log?
    • Time-based data
    • This data is everywhere!
    Server logs
    Twitter stream
    Financial transactions
    Metric / monitoring data
    ...
    • Log all things
    

    View Slide

40. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Why collect & centralize logs?
    • Access log files without system access
    • Shell scripting: Too limited or slow
    • Using unique ids for errors, aggregate it across
    your stack
    • Reporting (everyone can create his/her own report)
    • Tip: Unify your data to make it easily searchable
    

    View Slide

41. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Logstash architecture
    Logstash
    Input Output
    Filter
    ? ?
    collect and split alter and enrich store and visualize
    

    View Slide

42. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Inputs
    • Monitoring: collectd, graphite, ganglia, snmptrap,
    zenoss
    • Datastores: elasticsearch, redis, sqlite, s3
    • Queues: rabbitmq, zeromq
    • Logging: eventlog, lumberjack, gelf, log4j, relp,
    syslog, varnish log
    • Platforms: drupal_dblog, gemfire, heroku, sqs, s3,
    twitter
    • Local: exec, generator, file, stdin, pipe, unix
    • Protocol: imap, irc, stomp, tcp, udp, websocket,
    wmi, xmpp
    

    View Slide

43. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Filters
    • alter, anonymize, checksum, csv, drop, multiline
    • dns, date, extractnumbers, geoip, i18n, kv, noop,
    ruby, range
    • json, urldecode, useragent
    • metrics, sleep
    • … many, many more …
    

    View Slide

44. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Outputs
    • Store: elasticsearch, gemfire, mongodb, redis, riak,
    rabbitmq
    • Monitoring: ganglia, graphite, graphtastic, nagios,
    opentsdb, statsd, zabbix
    • Notification: email, hipchat, irc, pagerduty, sns
    • Protocol: gelf, http, lumberjack, metriccatcher, stomp,
    tcp, udp, websocket, xmpp
    • External Monitoring: boundary, circonus, cloudwatch,
    datadog, librato
    • External service: google big query, google cloud
    storage, jira, loggly, riemann, s3, sqs, syslog, zeromq
    • Local: csv, exec, file, pipe, stdout, null
    

    View Slide

45. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Demo
    

    View Slide

46. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Visualize with Kibana
    

    View Slide

47. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Kibana in 10 seconds
    • Visualize data in Elasticsearch
    • See real-time updates to the data
    • Build custom charts and dashboards
    • Apache License 2.0
    • Runs in browser (Chrome, FF, IE, Safari)
    • Part of Elasticsearch family
    

    View Slide

48. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    Useful helpers
    • Curator: index management
    http://www.elasticsearch.org/blog/curator-tending-your-time-series-indices/
    • Puppet module
    https://github.com/elasticsearch/puppet-logstash
    • Logstash forwarder: low overhead collector
    https://github.com/elasticsearch/logstash-forwarder
    • Logstash cookbook
    http://cookbook.logstash.net/
    

    View Slide

49. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    More info
    • Github: https://github.com/elasticsearch
    Code, issues there
    • Mailing lists
    Google groups, logstash-users and elasticsearch
    • IRC channels
    #logstash and #elasticsearch on freenode
    • We’re hiring!
    jobs@elasticsearch.com
    

    View Slide