Lock in $30 Savings on PRO—Offer Ends Soon! ⏳

ELK? Is that some kind of reindeer?

ELK? Is that some kind of reindeer?

Presentation for KWAN SmallTalks event in Lisbon by João Duarte. An overview on the ELK stack and the problems it addresses. Demo not included. http://www.meetup.com/KWAN-SmallTalks/events/220599708/

Elasticsearch Inc

March 03, 2015
Tweet

More Decks by Elasticsearch Inc

Other Decks in Technology

Transcript

  1. elasticsearch. João Duarte Software developer (J)Ruby lover Functional Programming Enthusiast

    Event Driven Architectures <3 Dog crazy person Chef $ whoami
  2. elasticsearch. Business questions Where are my customers coming from? How

    successful is our marketing campaign? Which products are trending in the busiest cities?
  3. elasticsearch. Operational questions Why is our database slow? When can

    we schedule a maintenance? Where are the attacks coming from?
  4. elasticsearch. but…what is a log? 66.249.73.185 - - [16/Feb/2014:09:47:54 -0500]

    "GET / HTTP/1.1" 200 37932 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http:// www.google.com/bot.html)"
  5. elasticsearch. but…what is a log? Oct 19 17:30:08 joaos-mbp.lan storeagent[270]

    <Critical>: SoftwareMap: Software map rebuild took 0.0037 seconds for 10 records and produced: { "/Applications/GarageBand.app" = "<CKSoftwareProduct: 0x7fa46b423330>: (com.apple.garageband10, 10.0.3, 682658836:513162807 VPP:NO source:LaunchPad /Applications/GarageBand.app) "; "/Applications/Install OS X Yosemite.app" = "<CKSoftwareProduct: 0x7fa46b73fbf0>: (com.apple.InstallAssistant.Yosemite, 1.6.5, 915041082:736932902 VPP:NO source:LaunchPad /Applications/ Install OS X Yosemite.app) "; "/Applications/Keynote.app" = "<CKSoftwareProduct: 0x7fa46b423990>: (com.apple.iWork.Keynote, 6.2.2, 409183694:662933456 VPP:NO source:LaunchPad /Applications/Keynote.app) "; "/Applications/Numbers.app" = "<CKSoftwareProduct: 0x7fa46b423140>: (com.apple.iWork.Numbers, 3.2.2, 409203825:662933458 VPP:NO source:LaunchPad /Applications/Numbers.app) "; "/Applications/Pages.app" = "<CKSoftwareProduct: 0x7fa46b423720>: (com.apple.iWork.Pages, 5.2.2, 409201541:662933457 VPP:NO source:LaunchPad /Applications/Pages.app) "; "/Applications/Pocket.app" = "<CKSoftwareProduct: 0x7fa46b423e10>: (com.readitlater.PocketMac, 1.6, 568494494:747482667 VPP:NO source:LaunchPad /Applications/Pocket.app) "; "/Applications/Twitter.app" = "<CKSoftwareProduct: 0x7fa46b423c80>: (com.twitter.twitter-mac, 3.1.0, 409789998:721812766 VPP:NO source:LaunchPad /Applications/Twitter.app) "; "/Applications/Wunderlist.app" = "<CKSoftwareProduct: 0x7fa46b423b00>: (com.wunderkinder.wunderlistdesktop, 3.1.0, 410628904:736952865 VPP:NO source:LaunchPad /Applications/ Wunderlist.app) "; "/Applications/iMovie.app" = "<CKSoftwareProduct: 0x7fa46b4235b0>: (com.apple.iMovieApp, 10.0.5, 408981434:668802749 VPP:NO source:LaunchPad /Applications/iMovie.app) "; "/Applications/iPhoto.app" = "<CKSoftwareProduct: 0x7fa46b4237d0>: (com.apple.iPhoto, 9.5.1, 408981381:233642646 VPP:NO source:LaunchPad /Applications/iPhoto.app) "; }
  6. elasticsearch. but…what is a log? Captain's Log, Stardate 43349.2. An

    unidentified distress signal has led to the discovery of a crashed Romulan vessel on the surface of Galorndon Core, a Federation planet. We have recovered one survivor, but Lieutenant Commander La Forge did not report back with the away team and is still missing.
  7. elasticsearch. Other problems with logs Multiple time formats Spread across

    all infrastructure Access and ability to understand logs is often restricted to devs/ops team
  8. elasticsearch. Logstash - Pipeline inputs filters outputs file syslog tcp

    websockets grok date split geoip anonymize elasticsearch pagerduty file
  9. elasticsearch. Logstash - Pipeline Stages input fetch/receive log data and

    generate events filter transform/enrich/normalize events output ship events towards other systems
  10. elasticsearch. input { } filter { } output { }

    Logstash - Pipeline Configuration
  11. elasticsearch. input { file { path => "/var/log/messages" } tcp

    { port => 3342 } } filter { grok { .. } date { .. } } output { elasticsearch { host => "localhost" } } Logstash - Pipeline Configuration
  12. elasticsearch. Logstash - Inputs tcp udp file syslog zeromq stdin

    rabbitmq redis S3 twitter websocket elasticsearch
  13. elasticsearch. Logstash - Field References output { email { to

    => "[email protected]" from => "[email protected]" subject => "Something bad happened" body => "The message: %{message}" } }
  14. elasticsearch. Logstash - Conditionals output { if [type] == "apache"

    { if [response] =~ /^5\d\d/ { nagios {...} } else if [response] =~ /^4\d\d/ { elasticsearch {...} } statsd { increment => "apache.%{status}" } } }
  15. elasticsearch. Logstash - Conditionals output { if [type] == "apache"

    { if [response] =~ /^5\d\d/ { pagerduty {...} } else if [response] =~ /^4\d\d/ { elasticsearch {...} } statsd { increment => "apache.%{status}" } } }
  16. elasticsearch. Elasticsearch HTTP & JSON, schema-less and document oriented written

    in Java using Lucene instances as unit of distribution distributed, built for horizontal scaling near-realtime search open source: Apache License 2.0
  17. elasticsearch. Elasticsearch heavily HTTP API driven HTTP port 9200 Transport

    port 9300 API example: curl -XGET 'http://localhost:9200/twitter/tweet/1'
  18. elasticsearch. Elasticsearch - Concepts cluster: one or more nodes with

    the same cluster name. Only one master node at any given time, chosen automatically; node: single elasticsearch instance; index: collection of documents (similar to database table). Maps to one or more shards; shard: an Apache Lucene instance. Basic unit of distribution in the cluster. There are primary and replica shards.
  19. elasticsearch. Elasticsearch - Document { "_index": "logstash-2014.03.11", "_type": "logs", "_id":

    "2CWDcFYVQFuj1MmORXB3pQ", "_source": { "message": "hello!", "@version": "1", "@timestamp": "2014-03-11T21:27:03.000Z", "host": "cadenza", "clientip": "121.99.231.187" } }
  20. elasticsearch. Defines how documents should be handled By default, generic

    mapping definition are applied to types Allows dynamic addition of unmapped fields Index templates allow you to automatically apply a mapping to a new index Mappings API for CRUD. To get a mapping for an index: Elasticsearch - Mappings $ curl http://localhost:9200/logstash-2013/_mapping
  21. elasticsearch. Elasticsearch - Document APIs Index - Index (store) a

    document Get - Retrieve a single document by its id Update - Modify an already indexed document Delete - Delete a document by its id Bulk Index - Index multiple documents in one request, which increases efficiency. The optimal number of documents depends on the particular cluster and use case And... Search - Explore the data
  22. elasticsearch. Elasticsearch - Document APIs $ curl -XPOST 'http://localhost:9200/twitter/tweet/' -d

    '{ "user" : "kimchy", "post_date" : "2009-11-15T14:12:12", "message" : "trying out Elasticsearch" }' { "_index" : "twitter", "_type" : "tweet", "_id" : "6a8ca01c-7896-48e9-81cc-9f70661fcb32", "_version" : 1, "created" : true }
  23. elasticsearch. Elasticsearch - Document APIs $ curl -XGET 'http://localhost:9200/twitter/tweet/1' {

    "_index" : "twitter", "_type" : "tweet", "_id" : "1", "_version" : 1, "found": true, "_source" : { "user" : "kimchy", "postDate" : "2009-11-15T14:12:12", "message" : "trying out Elasticsearch" } }
  24. elasticsearch. Elasticsearch - Document APIs $ curl -XDELETE 'http://localhost:9200/twitter/tweet/1' $

    curl -XHEAD -i 'http://localhost:9200/twitter/tweet/1' $ curl -XPOST 'localhost:9200/test/type1/1/_update' -d '{ "script" : "ctx._source.counter += count", "params" : { "count" : 4 } }’
  25. elasticsearch. Elasticsearch - Search $ curl -XGET 'http://localhost:9200/twitter/_search?q=user:kimchy' $ curl

    -XGET 'http://localhost:9200/kimchy,elasticsearch/tweet/ _search?q=tag:wow' $ curl -XGET 'http://localhost:9200/_search?q=tag:wow' $ curl -XGET 'http://localhost:9200/logstash-*/_count?pretty' { "count" : 1, "_shards" : { "total" : 5, "successful" : 5, "failed" : 0 } }
  26. elasticsearch. Elasticsearch - Query DSL $ curl -XGET 'http://localhost:9200/twitter/tweet/_search' -d

    '{ "query": { "filtered" : { "query" : { "query_string" : { "query" : "some query string here" } }, "filter" : { "term" : { "user" : "kimchy" } } } } }'
  27. elasticsearch. Elasticsearch - Aggregations $ curl -XGET 'http://localhost:9200/logstash-*/_search' -d '{

    "size": 0, "aggs" : { "top_uris" : { "terms" : { "field" : "clientip", "size" : 3, "exclude" : "128.30.28.58" } } } }'
  28. elasticsearch. Kibana Logstash: Process, structure, transport. Elasticsearch: Store, search, analyze.

    Kibana: Explore and visualize. NOTE: Kibana 4 was explained and shown during demo
  29. elasticsearch. Where to find us? Support: http://www.elasticsearch.com/support Community Resources: irc:

    #logstash and #elasticsearch on freenode email: [email protected] email: [email protected] meetups: http://elasticsearch.meetup.com/ twitter: @elasticsearch github: https://github.com/elasticsearch/