ELK? Is that some kind of reindeer?

Presentation for KWAN SmallTalks event in Lisbon by João Duarte. An overview on the ELK stack and the problems it addresses. Demo not included. http://www.meetup.com/KWAN-SmallTalks/events/220599708/

Transcript

1. elasticsearch. ELK? Is that some kind of reindeer? João Duarte
2. elasticsearch. Agenda Who am I? Problem? Logstash Elasticsearch Kibana Demo
3. elasticsearch. $ whoami
4. elasticsearch. João Duarte Software developer (J)Ruby lover Functional Programming Enthusiast Event Driven Architectures <3 Dog crazy person Chef $ whoami
5. elasticsearch. How awesome am I?
6. elasticsearch. How awesome am I? I left my old company ☹ sorry
7. elasticsearch.
8. elasticsearch.
9. elasticsearch. What’s the problem?
10. elasticsearch. Business questions Where are my customers coming from? How successful is our marketing campaign? Which products are trending in the busiest cities?
11. elasticsearch. Operational questions Why is our database slow? When can we schedule a maintenance? Where are the attacks coming from?
12. elasticsearch. It’s all in the logs!
13. elasticsearch. but…what is a log? 66.249.73.185 - - [16/Feb/2014:09:47:54 -0500] "GET / HTTP/1.1" 200 37932 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http:// www.google.com/bot.html)"
14. elasticsearch. but…what is a log? cpu_usage 0.74 1395532585
15. elasticsearch. but…what is a log? Oct 19 17:30:08 joaos-mbp.lan storeagent[270] <Critical>: SoftwareMap: Software map rebuild took 0.0037 seconds for 10 records and produced: { "/Applications/GarageBand.app" = "<CKSoftwareProduct: 0x7fa46b423330>: (com.apple.garageband10, 10.0.3, 682658836:513162807 VPP:NO source:LaunchPad /Applications/GarageBand.app) "; "/Applications/Install OS X Yosemite.app" = "<CKSoftwareProduct: 0x7fa46b73fbf0>: (com.apple.InstallAssistant.Yosemite, 1.6.5, 915041082:736932902 VPP:NO source:LaunchPad /Applications/ Install OS X Yosemite.app) "; "/Applications/Keynote.app" = "<CKSoftwareProduct: 0x7fa46b423990>: (com.apple.iWork.Keynote, 6.2.2, 409183694:662933456 VPP:NO source:LaunchPad /Applications/Keynote.app) "; "/Applications/Numbers.app" = "<CKSoftwareProduct: 0x7fa46b423140>: (com.apple.iWork.Numbers, 3.2.2, 409203825:662933458 VPP:NO source:LaunchPad /Applications/Numbers.app) "; "/Applications/Pages.app" = "<CKSoftwareProduct: 0x7fa46b423720>: (com.apple.iWork.Pages, 5.2.2, 409201541:662933457 VPP:NO source:LaunchPad /Applications/Pages.app) "; "/Applications/Pocket.app" = "<CKSoftwareProduct: 0x7fa46b423e10>: (com.readitlater.PocketMac, 1.6, 568494494:747482667 VPP:NO source:LaunchPad /Applications/Pocket.app) "; "/Applications/Twitter.app" = "<CKSoftwareProduct: 0x7fa46b423c80>: (com.twitter.twitter-mac, 3.1.0, 409789998:721812766 VPP:NO source:LaunchPad /Applications/Twitter.app) "; "/Applications/Wunderlist.app" = "<CKSoftwareProduct: 0x7fa46b423b00>: (com.wunderkinder.wunderlistdesktop, 3.1.0, 410628904:736952865 VPP:NO source:LaunchPad /Applications/ Wunderlist.app) "; "/Applications/iMovie.app" = "<CKSoftwareProduct: 0x7fa46b4235b0>: (com.apple.iMovieApp, 10.0.5, 408981434:668802749 VPP:NO source:LaunchPad /Applications/iMovie.app) "; "/Applications/iPhoto.app" = "<CKSoftwareProduct: 0x7fa46b4237d0>: (com.apple.iPhoto, 9.5.1, 408981381:233642646 VPP:NO source:LaunchPad /Applications/iPhoto.app) "; }
16. elasticsearch. but…what is a log? Captain's Log, Stardate 43349.2. An unidentified distress signal has led to the discovery of a crashed Romulan vessel on the surface of Galorndon Core, a Federation planet. We have recovered one survivor, but Lieutenant Commander La Forge did not report back with the away team and is still missing.
17. elasticsearch. Other problems with logs Multiple time formats Spread across all infrastructure Access and ability to understand logs is often restricted to devs/ops team
18. elasticsearch. Elasticsearch - ELK servers Logstash elasticsearch Kibana
19. elasticsearch. Logstash
20. elasticsearch. process Logstash
21. elasticsearch. Logstash Event processing pipeline Stream oriented Plugin based Written in Ruby Open Source
22. elasticsearch. Jordan Sissel (@jordansissel)
23. elasticsearch. Logstash - Pipeline inputs filters outputs file syslog tcp websockets grok date split geoip anonymize elasticsearch pagerduty file
24. elasticsearch. Logstash - Pipeline Stages input fetch/receive log data and generate events filter transform/enrich/normalize events output ship events towards other systems
25. elasticsearch. input { } filter { } output { } Logstash - Pipeline Configuration
26. elasticsearch. input { file { path => "/var/log/messages" } tcp { port => 3342 } } filter { grok { .. } date { .. } } output { elasticsearch { host => "localhost" } } Logstash - Pipeline Configuration
27. elasticsearch. Logstash - Inputs tcp udp file syslog zeromq stdin rabbitmq redis S3 twitter websocket elasticsearch
28. elasticsearch. Logstash - Filters grok date geoip split drop mutate kv anonymize zeromq elasticsearch
29. elasticsearch. Logstash - Outputs elasticsearch file tcp udp stdout S3 HTTP rabbitmq redis pagerduty
30. elasticsearch. Logstash - Field References output { email { to => "somebody@example.com" from => "logstash@example.com" subject => "Something bad happened" body => "The message: %{message}" } }
31. elasticsearch. Logstash - Conditionals output { if [type] == "apache" { if [response] =~ /^5\d\d/ { nagios {...} } else if [response] =~ /^4\d\d/ { elasticsearch {...} } statsd { increment => "apache.%{status}" } } }
32. elasticsearch. Logstash - Conditionals output { if [type] == "apache" { if [response] =~ /^5\d\d/ { pagerduty {...} } else if [response] =~ /^4\d\d/ { elasticsearch {...} } statsd { increment => "apache.%{status}" } } }
33. elasticsearch. Elasticsearch
34. elasticsearch. Elasticsearch search
35. elasticsearch. Elasticsearch HTTP & JSON, schema-less and document oriented written in Java using Lucene instances as unit of distribution distributed, built for horizontal scaling near-realtime search open source: Apache License 2.0
36. elasticsearch. Elasticsearch heavily HTTP API driven HTTP port 9200 Transport port 9300 API example: curl -XGET 'http://localhost:9200/twitter/tweet/1'
37. elasticsearch. Elasticsearch - Concepts cluster: one or more nodes with the same cluster name. Only one master node at any given time, chosen automatically; node: single elasticsearch instance; index: collection of documents (similar to database table). Maps to one or more shards; shard: an Apache Lucene instance. Basic unit of distribution in the cluster. There are primary and replica shards.
38. elasticsearch. Elasticsearch - Nodes and Shards
39. elasticsearch. Elasticsearch - Document { "_index": "logstash-2014.03.11", "_type": "logs", "_id": "2CWDcFYVQFuj1MmORXB3pQ", "_source": { "message": "hello!", "@version": "1", "@timestamp": "2014-03-11T21:27:03.000Z", "host": "cadenza", "clientip": "121.99.231.187" } }
40. elasticsearch. Defines how documents should be handled By default, generic mapping definition are applied to types Allows dynamic addition of unmapped fields Index templates allow you to automatically apply a mapping to a new index Mappings API for CRUD. To get a mapping for an index: Elasticsearch - Mappings $ curl http://localhost:9200/logstash-2013/_mapping
41. elasticsearch. Elasticsearch - Document APIs Index - Index (store) a document Get - Retrieve a single document by its id Update - Modify an already indexed document Delete - Delete a document by its id Bulk Index - Index multiple documents in one request, which increases efficiency. The optimal number of documents depends on the particular cluster and use case And... Search - Explore the data
42. elasticsearch. Elasticsearch - Document APIs $ curl -XPOST 'http://localhost:9200/twitter/tweet/' -d '{ "user" : "kimchy", "post_date" : "2009-11-15T14:12:12", "message" : "trying out Elasticsearch" }' { "_index" : "twitter", "_type" : "tweet", "_id" : "6a8ca01c-7896-48e9-81cc-9f70661fcb32", "_version" : 1, "created" : true }
43. elasticsearch. Elasticsearch - Document APIs $ curl -XGET 'http://localhost:9200/twitter/tweet/1' { "_index" : "twitter", "_type" : "tweet", "_id" : "1", "_version" : 1, "found": true, "_source" : { "user" : "kimchy", "postDate" : "2009-11-15T14:12:12", "message" : "trying out Elasticsearch" } }
44. elasticsearch. Elasticsearch - Document APIs $ curl -XDELETE 'http://localhost:9200/twitter/tweet/1' $ curl -XHEAD -i 'http://localhost:9200/twitter/tweet/1' $ curl -XPOST 'localhost:9200/test/type1/1/_update' -d '{ "script" : "ctx._source.counter += count", "params" : { "count" : 4 } }’
45. elasticsearch. Elasticsearch - Search $ curl -XGET 'http://localhost:9200/twitter/_search?q=user:kimchy' $ curl -XGET 'http://localhost:9200/kimchy,elasticsearch/tweet/ _search?q=tag:wow' $ curl -XGET 'http://localhost:9200/_search?q=tag:wow' $ curl -XGET 'http://localhost:9200/logstash-*/_count?pretty' { "count" : 1, "_shards" : { "total" : 5, "successful" : 5, "failed" : 0 } }
46. elasticsearch. Elasticsearch - Query DSL $ curl -XGET 'http://localhost:9200/twitter/tweet/_search' -d '{ "query": { "filtered" : { "query" : { "query_string" : { "query" : "some query string here" } }, "filter" : { "term" : { "user" : "kimchy" } } } } }'
47. elasticsearch. Elasticsearch - Aggregations $ curl -XGET 'http://localhost:9200/logstash-*/_search' -d '{ "size": 0, "aggs" : { "top_uris" : { "terms" : { "field" : "clientip", "size" : 3, "exclude" : "128.30.28.58" } } } }'
48. elasticsearch. Architecture
49. elasticsearch. Architecture
50. elasticsearch. Architecture
51. elasticsearch. Kibana
52. elasticsearch. Kibana visualize
53. elasticsearch. Kibana Logstash: Process, structure, transport. Elasticsearch: Store, search, analyze. Kibana: Explore and visualize. NOTE: Kibana 4 was explained and shown during demo
54. elasticsearch. demo
55. elasticsearch. demo
56. elasticsearch. demo
57. elasticsearch. Where to find us? Support: http://www.elasticsearch.com/support Community Resources: irc: #logstash and #elasticsearch on freenode email: logstash-users@googlegroups.com email: elasticsearch-users@googlegroups.com meetups: http://elasticsearch.meetup.com/ twitter: @elasticsearch github: https://github.com/elasticsearch/
58. elasticsearch. Anything else?
59. elasticsearch. Thank you! Questions? @jsvd