Upgrade to Pro — share decks privately, control downloads, hide ads and more …

ELK? Is that some kind of reindeer?

ELK? Is that some kind of reindeer?

Presentation for KWAN SmallTalks event in Lisbon by João Duarte. An overview on the ELK stack and the problems it addresses. Demo not included. http://www.meetup.com/KWAN-SmallTalks/events/220599708/

Elasticsearch Inc

March 03, 2015
Tweet

More Decks by Elasticsearch Inc

Other Decks in Technology

Transcript

  1. elasticsearch. ELK? Is that some kind of reindeer? João Duarte

  2. elasticsearch. Agenda Who am I? Problem? Logstash Elasticsearch Kibana Demo

  3. elasticsearch. $ whoami

  4. elasticsearch. João Duarte Software developer (J)Ruby lover Functional Programming Enthusiast

    Event Driven Architectures <3 Dog crazy person Chef $ whoami
  5. elasticsearch. How awesome am I?

  6. elasticsearch. How awesome am I? I left my old company

    ☹ sorry
  7. elasticsearch.

  8. elasticsearch.

  9. elasticsearch. What’s the problem?

  10. elasticsearch. Business questions Where are my customers coming from? How

    successful is our marketing campaign? Which products are trending in the busiest cities?
  11. elasticsearch. Operational questions Why is our database slow? When can

    we schedule a maintenance? Where are the attacks coming from?
  12. elasticsearch. It’s all in the logs!

  13. elasticsearch. but…what is a log? 66.249.73.185 - - [16/Feb/2014:09:47:54 -0500]

    "GET / HTTP/1.1" 200 37932 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http:// www.google.com/bot.html)"
  14. elasticsearch. but…what is a log? cpu_usage 0.74 1395532585

  15. elasticsearch. but…what is a log? Oct 19 17:30:08 joaos-mbp.lan storeagent[270]

    <Critical>: SoftwareMap: Software map rebuild took 0.0037 seconds for 10 records and produced: { "/Applications/GarageBand.app" = "<CKSoftwareProduct: 0x7fa46b423330>: (com.apple.garageband10, 10.0.3, 682658836:513162807 VPP:NO source:LaunchPad /Applications/GarageBand.app) "; "/Applications/Install OS X Yosemite.app" = "<CKSoftwareProduct: 0x7fa46b73fbf0>: (com.apple.InstallAssistant.Yosemite, 1.6.5, 915041082:736932902 VPP:NO source:LaunchPad /Applications/ Install OS X Yosemite.app) "; "/Applications/Keynote.app" = "<CKSoftwareProduct: 0x7fa46b423990>: (com.apple.iWork.Keynote, 6.2.2, 409183694:662933456 VPP:NO source:LaunchPad /Applications/Keynote.app) "; "/Applications/Numbers.app" = "<CKSoftwareProduct: 0x7fa46b423140>: (com.apple.iWork.Numbers, 3.2.2, 409203825:662933458 VPP:NO source:LaunchPad /Applications/Numbers.app) "; "/Applications/Pages.app" = "<CKSoftwareProduct: 0x7fa46b423720>: (com.apple.iWork.Pages, 5.2.2, 409201541:662933457 VPP:NO source:LaunchPad /Applications/Pages.app) "; "/Applications/Pocket.app" = "<CKSoftwareProduct: 0x7fa46b423e10>: (com.readitlater.PocketMac, 1.6, 568494494:747482667 VPP:NO source:LaunchPad /Applications/Pocket.app) "; "/Applications/Twitter.app" = "<CKSoftwareProduct: 0x7fa46b423c80>: (com.twitter.twitter-mac, 3.1.0, 409789998:721812766 VPP:NO source:LaunchPad /Applications/Twitter.app) "; "/Applications/Wunderlist.app" = "<CKSoftwareProduct: 0x7fa46b423b00>: (com.wunderkinder.wunderlistdesktop, 3.1.0, 410628904:736952865 VPP:NO source:LaunchPad /Applications/ Wunderlist.app) "; "/Applications/iMovie.app" = "<CKSoftwareProduct: 0x7fa46b4235b0>: (com.apple.iMovieApp, 10.0.5, 408981434:668802749 VPP:NO source:LaunchPad /Applications/iMovie.app) "; "/Applications/iPhoto.app" = "<CKSoftwareProduct: 0x7fa46b4237d0>: (com.apple.iPhoto, 9.5.1, 408981381:233642646 VPP:NO source:LaunchPad /Applications/iPhoto.app) "; }
  16. elasticsearch. but…what is a log? Captain's Log, Stardate 43349.2. An

    unidentified distress signal has led to the discovery of a crashed Romulan vessel on the surface of Galorndon Core, a Federation planet. We have recovered one survivor, but Lieutenant Commander La Forge did not report back with the away team and is still missing.
  17. elasticsearch. Other problems with logs Multiple time formats Spread across

    all infrastructure Access and ability to understand logs is often restricted to devs/ops team
  18. elasticsearch. Elasticsearch - ELK servers Logstash elasticsearch Kibana

  19. elasticsearch. Logstash

  20. elasticsearch. process Logstash

  21. elasticsearch. Logstash Event processing pipeline Stream oriented Plugin based Written

    in Ruby Open Source
  22. elasticsearch. Jordan Sissel (@jordansissel)

  23. elasticsearch. Logstash - Pipeline inputs filters outputs file syslog tcp

    websockets grok date split geoip anonymize elasticsearch pagerduty file
  24. elasticsearch. Logstash - Pipeline Stages input fetch/receive log data and

    generate events filter transform/enrich/normalize events output ship events towards other systems
  25. elasticsearch. input { } filter { } output { }

    Logstash - Pipeline Configuration
  26. elasticsearch. input { file { path => "/var/log/messages" } tcp

    { port => 3342 } } filter { grok { .. } date { .. } } output { elasticsearch { host => "localhost" } } Logstash - Pipeline Configuration
  27. elasticsearch. Logstash - Inputs tcp udp file syslog zeromq stdin

    rabbitmq redis S3 twitter websocket elasticsearch
  28. elasticsearch. Logstash - Filters grok date geoip split drop mutate

    kv anonymize zeromq elasticsearch
  29. elasticsearch. Logstash - Outputs elasticsearch file tcp udp stdout S3

    HTTP rabbitmq redis pagerduty
  30. elasticsearch. Logstash - Field References output { email { to

    => "[email protected]" from => "[email protected]" subject => "Something bad happened" body => "The message: %{message}" } }
  31. elasticsearch. Logstash - Conditionals output { if [type] == "apache"

    { if [response] =~ /^5\d\d/ { nagios {...} } else if [response] =~ /^4\d\d/ { elasticsearch {...} } statsd { increment => "apache.%{status}" } } }
  32. elasticsearch. Logstash - Conditionals output { if [type] == "apache"

    { if [response] =~ /^5\d\d/ { pagerduty {...} } else if [response] =~ /^4\d\d/ { elasticsearch {...} } statsd { increment => "apache.%{status}" } } }
  33. elasticsearch. Elasticsearch

  34. elasticsearch. Elasticsearch search

  35. elasticsearch. Elasticsearch HTTP & JSON, schema-less and document oriented written

    in Java using Lucene instances as unit of distribution distributed, built for horizontal scaling near-realtime search open source: Apache License 2.0
  36. elasticsearch. Elasticsearch heavily HTTP API driven HTTP port 9200 Transport

    port 9300 API example: curl -XGET 'http://localhost:9200/twitter/tweet/1'
  37. elasticsearch. Elasticsearch - Concepts cluster: one or more nodes with

    the same cluster name. Only one master node at any given time, chosen automatically; node: single elasticsearch instance; index: collection of documents (similar to database table). Maps to one or more shards; shard: an Apache Lucene instance. Basic unit of distribution in the cluster. There are primary and replica shards.
  38. elasticsearch. Elasticsearch - Nodes and Shards

  39. elasticsearch. Elasticsearch - Document { "_index": "logstash-2014.03.11", "_type": "logs", "_id":

    "2CWDcFYVQFuj1MmORXB3pQ", "_source": { "message": "hello!", "@version": "1", "@timestamp": "2014-03-11T21:27:03.000Z", "host": "cadenza", "clientip": "121.99.231.187" } }
  40. elasticsearch. Defines how documents should be handled By default, generic

    mapping definition are applied to types Allows dynamic addition of unmapped fields Index templates allow you to automatically apply a mapping to a new index Mappings API for CRUD. To get a mapping for an index: Elasticsearch - Mappings $ curl http://localhost:9200/logstash-2013/_mapping
  41. elasticsearch. Elasticsearch - Document APIs Index - Index (store) a

    document Get - Retrieve a single document by its id Update - Modify an already indexed document Delete - Delete a document by its id Bulk Index - Index multiple documents in one request, which increases efficiency. The optimal number of documents depends on the particular cluster and use case And... Search - Explore the data
  42. elasticsearch. Elasticsearch - Document APIs $ curl -XPOST 'http://localhost:9200/twitter/tweet/' -d

    '{ "user" : "kimchy", "post_date" : "2009-11-15T14:12:12", "message" : "trying out Elasticsearch" }' { "_index" : "twitter", "_type" : "tweet", "_id" : "6a8ca01c-7896-48e9-81cc-9f70661fcb32", "_version" : 1, "created" : true }
  43. elasticsearch. Elasticsearch - Document APIs $ curl -XGET 'http://localhost:9200/twitter/tweet/1' {

    "_index" : "twitter", "_type" : "tweet", "_id" : "1", "_version" : 1, "found": true, "_source" : { "user" : "kimchy", "postDate" : "2009-11-15T14:12:12", "message" : "trying out Elasticsearch" } }
  44. elasticsearch. Elasticsearch - Document APIs $ curl -XDELETE 'http://localhost:9200/twitter/tweet/1' $

    curl -XHEAD -i 'http://localhost:9200/twitter/tweet/1' $ curl -XPOST 'localhost:9200/test/type1/1/_update' -d '{ "script" : "ctx._source.counter += count", "params" : { "count" : 4 } }’
  45. elasticsearch. Elasticsearch - Search $ curl -XGET 'http://localhost:9200/twitter/_search?q=user:kimchy' $ curl

    -XGET 'http://localhost:9200/kimchy,elasticsearch/tweet/ _search?q=tag:wow' $ curl -XGET 'http://localhost:9200/_search?q=tag:wow' $ curl -XGET 'http://localhost:9200/logstash-*/_count?pretty' { "count" : 1, "_shards" : { "total" : 5, "successful" : 5, "failed" : 0 } }
  46. elasticsearch. Elasticsearch - Query DSL $ curl -XGET 'http://localhost:9200/twitter/tweet/_search' -d

    '{ "query": { "filtered" : { "query" : { "query_string" : { "query" : "some query string here" } }, "filter" : { "term" : { "user" : "kimchy" } } } } }'
  47. elasticsearch. Elasticsearch - Aggregations $ curl -XGET 'http://localhost:9200/logstash-*/_search' -d '{

    "size": 0, "aggs" : { "top_uris" : { "terms" : { "field" : "clientip", "size" : 3, "exclude" : "128.30.28.58" } } } }'
  48. elasticsearch. Architecture

  49. elasticsearch. Architecture

  50. elasticsearch. Architecture

  51. elasticsearch. Kibana

  52. elasticsearch. Kibana visualize

  53. elasticsearch. Kibana Logstash: Process, structure, transport. Elasticsearch: Store, search, analyze.

    Kibana: Explore and visualize. NOTE: Kibana 4 was explained and shown during demo
  54. elasticsearch. demo

  55. elasticsearch. demo

  56. elasticsearch. demo

  57. elasticsearch. Where to find us? Support: http://www.elasticsearch.com/support Community Resources: irc:

    #logstash and #elasticsearch on freenode email: [email protected] email: [email protected] meetups: http://elasticsearch.meetup.com/ twitter: @elasticsearch github: https://github.com/elasticsearch/
  58. elasticsearch. Anything else?

  59. elasticsearch. Thank you! Questions? @jsvd