A deeper look at the ELK Stack: Elasticsearch, Logstash & Kibana

Slide 1

Slide 1 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Aaron Mildenstein (@theuntergeek) Elastic, Inc. A deeper look at the ELK Stack

Slide 2

Slide 2 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Lab Resources • https://goo.gl/EttDLR (https://s3.amazonaws.com/elk- workshop.elasticsearch.org/hands-on-workshop/20150509-openwest/hands-on- workshop.tar.gz) • USB Drive (hopefully there are enough to pass around)

Slide 3

Slide 3 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited We’ve clearly hit a nerve 13M+ cumulative downloads Million of Downloads 0 2 4 6 8 Oct'12 Jan'13 Apr'13 Jul'13 Oct'13 Jan'14 Apr'14 Nearly 5M downloads in the last year

Slide 4

Slide 4 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Growing Customer Base

Slide 5

Slide 5 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited The History of ELK

Slide 6

Slide 6 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited History of the ELK Stack • Logstash was started in 2009 by Jordan Sissel • Elasticsearch was ﬁrst released in 2010 by Shay Banon • Kibana was begun in 2011 by Rashid Khan • Elasticsearch (the company) was founded in 2012 • Rashid joined Elasticsearch in January, 2013 • Jordan joined Elasticsearch in August, 2013 • Much of the development on all three projects is now done in-house, in addition to open source contributions

Slide 7

Slide 7 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Search Use Cases

Slide 8

Slide 8 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Unstructured search

Slide 9

Slide 9 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Structured search www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited

Slide 10

Slide 10 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Enrichment www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited

Slide 11

Slide 11 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Sorting www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited

Slide 12

Slide 12 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Pagination www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited

Slide 13

Slide 13 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Aggregation www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited

Slide 14

Slide 14 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Suggestions www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited

Slide 15

Slide 15 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Elasticsearch

Slide 16

Slide 16 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited What is Elasticsearch? • Document-oriented search engine • JSON based (both document store and REST API) • Built on top of Apache Lucene • Schema Free / Schema-Less • Yet enables control of schema when needed (via mappings) • Distributed Model • Scales Up+Out, Highly Available • Multi-tenant data • Dynamically create/delete indices • API centric & RESTful • Most functionality + cluster statistics are exposed via API

Slide 17

Slide 17 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Basic glossary • Maecenas aliquam maecenas ligula nostra, accumsan taciti. Sociis mauris in integer • El eu libero cras interdum at eget habitasse elementum est, ipsum purus pede • Aliquet sed. Lorem ipsum dolor sit amet, ligula suspendisse nulla pretium, rhoncus cluster A cluster consists of one or more nodes which share the same cluster name. Each cluster has a single master node which is chosen automatically by the cluster and which can be replaced automatically if the current master node fails. node A node is a running instance of elasticsearch which belongs to a cluster. Multiple nodes can be started on a single server for testing purposes, but usually you should have one node per server. At startup, a node will use multicast (or unicast, if speciﬁed) to discover an existing cluster with the same cluster name and will try to join that cluster.

Slide 18

Slide 18 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Basic glossary • Maecenas aliquam maecenas ligula nostra, accumsan taciti. Sociis mauris in integer • El eu libero cras interdum at eget habitasse elementum est, ipsum purus pede • Aliquet sed. Lorem ipsum dolor sit amet, ligula suspendisse nulla pretium, rhoncus index An index can be seen as a named collection of documents. It is a logical namespace which maps to one or more primary shards and can have zero or more replica shards. shard A shard is a single Apache Lucene instance. It is a low- level “worker” unit which is managed automatically. Shards are distributed across all nodes in the cluster, and can move shards automatically from one node to another in the case of node failure, or the addition of new nodes. There are two types of shards: primary and replica.

Slide 19

Slide 19 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Basic glossary • Maecenas aliquam maecenas ligula nostra, accumsan taciti. Sociis mauris in integer • El eu libero cras interdum at eget habitasse elementum est, ipsum purus pede • Aliquet sed. Lorem ipsum dolor sit amet, ligula suspendisse nulla pretium, rhoncus Primary shard An index can have one or more primary shards (defaults to 5) and it is not possible to change this number after index creation. When you index a document, it is ﬁrst indexed on the primary shard, then on all replicas of this shard. Replica shard Each primary shard can have zero or more replicas (defaults to 1). A replica is a copy of the primary shard, and serves two purposes: ‣ Increase high availability - a replica is another copy of the data and will be promoted to a primary shard if the primary fails ‣ Increase performance - get and search requests can be handled by primary or replica shards

Slide 20

Slide 20 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Create Index API Creating Index a with 2 shards and 1 replica (a total of 4 shards) Creating Index b with 3 shards and 1 replica (a total of 6 shards) curl -XPUT 'localhost:9200/a' -d '{ "settings" : { "number_of_shards" : 2, "number_of_replicas" : 1 } }' curl -XPUT 'localhost:9200/b' -d '{ "settings" : { "number_of_shards" : 3, "number_of_replicas" : 1 } }'

Slide 21

Slide 21 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited curl -XPUT ‘localhost:9200/crunchbase/person/1’ -d '{ "first_name" : "Tony", "last_name" : "Stark" }' Index API target index name HTTP REST method document JSON source document type document id

Slide 22

Slide 22 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited • It is possible to retrieve a speciﬁc document from the index using its _type and _id • The GET operation is realtime Meaning, once a document is indexed, it is immediately available to be retrieved using the GET API curl -XGET 'localhost:9200/crunchbase/person/1' Get API target index name HTTP REST operation document type document id

Slide 23

Slide 23 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Exists API • Check if a document is in the index Without the overhead of loading it • The response is based on HTTP status code 200 (OK) if exists 404 (NOT FOUND) if doesn’t exist curl -XHEAD -i 'localhost:9200/crunchbase/person/1'

Slide 24

Slide 24 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Update API • Update by partial data Partial doc is merged with existing doc    Non-object properties with the same key are replaced. Object properties are recursively merged curl -XPOST 'localhost:9200/crunchbase/person/1/_update' -d '{ "doc" : { "first_name" : "Antonio" } }'

Slide 25

Slide 25 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited • Deleting a speciﬁc document by _id • Response 200 (OK) if deleted 404 if not found { "found" : true, "_index" : "test", "_type" : "person", "_id" : "1", "_version" : 3 } Delete API Indication if it was actually found curl -XDELETE 'localhost:9200/crunchbase/person/1'

Slide 26

Slide 26 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited • Sometimes you'd like to get multiple documents in one go Avoid round trips when using the Get API curl 'localhost:9200/_mget' -d '{ "docs" : [ { "_index" : "crunchbase", "_type" : "person", "_id" : "1" }, { "_index" : "marvels", "_type" : "hero", "_id" : "2" "_source" : [ "first_name" ] } ] }' Multi Get API index name document type document id Optionally specify what ﬁelds should be returned

Slide 27

Slide 27 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited • Minimizes round trips when performing bulk index/delete/ update operations • The format of a bulk request is as follows { "delete" : { "_index" : "crunchbase", "_type" : "person", "_id" : "2" } }\n { "index" : { "_index" : "crunchbase", "_type" : "person", "_id" : "1" } }\n { "first_name" : "Tony", "last_name" : "Stark" }\n . . . { "create" : { "_index" : "crunchbase", "_type" : "person", "_id" : "3" } }\n { "first_name" : "Thor", "last_name" : "Odinson" }\n Bulk API each line must end with a line break (incl. the last line) optional action body action metadata

Slide 28

Slide 28 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Analysis

Slide 29

Slide 29 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited

Slide 30

Slide 30 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Grep! • Which plays contain the word “darling” in the complete works of Shakespeare? ‣ Grep it! ‣ Go over each play, word by word, and mark the play that contains it • Linear to the number of words • Fails at large scale

Slide 31

Slide 31 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Inverted Index • Inverting Shakespeare ‣ Take all the plays and break them down word by word ‣ For each word, store the ids of the documents that contain it ‣ Sort all tokens (words) • Search First look for the relevant word (fast as words are sorted), if found, iterate over the document ids that are associated it

Slide 32

Slide 32 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited

Slide 33

Slide 33 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Term Doc 1 Doc 2 Doc 3 breathe brings buds but by can … damasked darling date day deaf death declines delight

Slide 34

Slide 34 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Analyzers • Analysis => Tokenization and normalization • Analyzers => Analysis and token filters • Token filters act on the token stream - can drop and modify existing tokens, or add new ones. • Out of the box, many analyzers are available — Standard analyzer, Whitespace analyzer, language analyzers • Can define/build custom analyzers

Slide 35

Slide 35 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited _analyze API GET /_analyze?analyzer=whitespace&text=FOO BAR { "tokens": [ { "token": "FOO", "start_offset": 0, "end_offset": 3, "type": "word", "position": 1 }, { "token": "BAR", "start_offset": 4, "end_offset": 7, "type": "word", "position": 2 } ]

Slide 36

Slide 36 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Rich Search via Query DSL • Queries • Unstructured search, enables to query the data based on textual analysis (free text search). Queries score documents by relevancy (supports powerful custom scoring algorithms). To name a few: • match ‣ bool (boolean) ‣ histogram ‣ Filters • Structured search, enables narrowing the search context based on known document structure (no scoring and very fast). To name a few: ‣ term ‣ range ‣ bool (boolean)

Slide 37

Slide 37 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Querying Powerful and rich Query DSL Queries are analyzed too Near real time (from indexing to querying) GET /_search -d '{  {  "query": {  "match": { "tweet": "elasticsearch" }  } } 

Slide 38

Slide 38 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Results { "took": 15, "timed_out": false, "_shards": { "total": 5, "successful": 5, "failed": 0 }, "hits": { "total": 1, "max_score": 0.30685282, "hits": [ { "_index": "twitter", "_type": "tweets", "_id": "cxxV4_TST_iR2zH1GuedVQ", "_score": 0.30685282, "_source": { “awesome #logstash #kibana #elasticsearch presentation with real life use case demo by @webmat at @devopsmontreal" 

Slide 39

Slide 39 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Suggestions Look familiar?

Slide 40

Slide 40 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Analytics Un-invert the inverted index (Field data) Load the field data to memory Group By — popular terms, significant terms, ranges, dates, geolocation Metrics — count, min, max, sum, avg, percentiles, cardinality, Nested aggregations helps slice and dice data

Slide 41

Slide 41 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Tweets per month GET /_all/tweet/_search -d '  {  "aggs": {  "tweets_by_month": {  "date_histogram": {  "field": "date",  "interval": "month"  }  }  }  } 

Slide 42

Slide 42 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Aggregations Facets Free text search

Slide 43

Slide 43 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Aggregations Use aggregations to build analytics tools & dashboards

Slide 44

Slide 44 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Aggregations: Buckets & Metrics • Two categories of aggregations - Buckets & Metrics • Buckets Aggregations that build buckets. Each bucket is associated with some criteria over documents. During query execution, each document is evaluated against the created buckets and each bucket keeps track of what documents “fall” in it. Each bucket effectively defines a set of documents derived from the document set within the aggregations scope. • Metrics Aggregations that given a set of documents, produce a single/multiple scalar/s. Typically, metrics aggregations generate numeric stats that are computed over a specific document set

Slide 45

Slide 45 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Bucket - terms "aggregations": { "states": { "buckets": [ { "key": "ma", "doc_count": 841 }, { "key": "ca", "doc_count": 631 }, ... "key": "ny", "doc_count": 630 }, { "key": "nj", "doc_count": 560 }, { "key": "wa", "doc_count": 525 } ] } } Response wa 16% nj 18% ny 20% ca 20% ma 26%

Slide 46

Slide 46 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Bucket - *_range • date_range A dedicated range aggregations that works on date fields. Ranges can be defined as date math expressions • ip_range A dedicated range aggregation that works on ip fields. Ranges can be defined as ipv4 strings or CIDR masks { "from" : "now-1M", "to" : "now" } { "from" : "10.0.0.0", "to" : "10.0.0.128" } { "mask" : "10.0.0.0/25" }

Slide 47

Slide 47 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Bucket - histogram "aggregations": { "grades_distribution": { "buckets": [ { "key": 60, "doc_count": 467 }, { "key": 70, "doc_count": 873 }, { "key": 80, "doc_count": 930 }, { "key": 90, "doc_count": 915 } ] } } Response 0 250 500 750 1000 60 70 80 90 915 930 873 467 By default, only non-empty buckets will be returned

Slide 48

Slide 48 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Metrics - extended_stats The following computes statistics on student exam scores over a set of document (each representing an exam result) { "aggs" : { "grades" : { "extended_stats" : { "field" : "grade" } } } } "aggregations": { "grades": { "count": 4375, "min": 65, "max": 99, "avg": 82.14765714285714, "sum": 359396, "sum_of_squares": 29970052, "variance": 102.06002593959144, "std_deviation": 10.102476228113158 } } Request Response { "subject": "Mathematics", "state": "CA", "age": 8, "grade": 69, "male": true } Sample document

Slide 49

Slide 49 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Aggregations • Enables slicing & dicing the data • Provides multi-dimensional grouping of results. e.g. Top URLs by country. • Many types available • All operate over values extracted from the documents - usually from specific fields of the documents, but highly customizable using scripts ‣ terms ‣ range / date_range / ip_range ‣ geo_distance / geohash_grid ‣ histogram / date_histogram ‣ stats / avg / max / min / sum / percentiles

Slide 50

Slide 50 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Our Logs Have Issues

Slide 51

Slide 51 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Problem 1: No Consistency • Every application and device logs in its own special way. • Expert in each log format required to use the logs. • Difficult to search across because of this formatting problem.

Slide 52

Slide 52 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited No Consistency 120707 0:40:34 4 Connect root@localhost on 4 Query select @@version_comment limit 1 120707 0:40:45 4 Query select * from mysql.user 120707 0:41:18 5 Query hello world

Slide 53

Slide 53 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited No Consistency 120707 0:37:09 [Note] Plugin 'FEDERATED' is disabled. 120707 0:37:09 InnoDB: The InnoDB memory heap is disabled 120707 0:37:09 InnoDB: Mutexes and rw_locks use GCC atomic builtins 120707 0:37:09 InnoDB: Compressed tables use zlib 1.2.5 120707 0:37:09 InnoDB: Using Linux native AIO 120707 0:37:09 InnoDB: Initializing buffer pool, size = 128.0M 120707 0:37:09 InnoDB: Completed initialization of buffer pool

Slide 54

Slide 54 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited No Consistency # User@Host: biz_1[biz_1] @ localhost [] # Query_time: 0.000273 Lock_time: 0.000104 Rows_sent: 1 Rows_examined: 1 SET timestamp=1255345490; SELECT * FROM organization_details;

Slide 55

Slide 55 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited No Consistency Mar 23 22:05:24 Macintosh com.apple.launchd[1] (httpd): Throttling respawn: Will start in 10 seconds

Slide 56

Slide 56 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Problem 2: Time Formats 130460505 Oct 11 20:21:47 [29/Apr/2011:07:05:26 +0000] 020805 13:51:24 @4000000037c219bf2ef02e94

Slide 57

Slide 57 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Problem 3: Decentralized • Logs are spread across all of your servers • Many servers have many different kinds of logs • ssh + grep aren’t scalable

Slide 58

Slide 58 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Problem 4: Experts Required • People interested in the logs often… • Do not have access to read the logs. • Do not have expertise to understand the data. • Do not know where the logs are.

Slide 59

Slide 59 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Logstash

Slide 60

Slide 60 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited What is Logstash? • Event processing engine, optimized for logs • raw data in, enriched data out • Written in Ruby, runs on JRuby • Simple to extend, efficient to run • Events pass through a pipeline • Inputs: receive data from files, network, etc. • Filters: enrich, massage, process the event data • Outputs: send event data to other systems • Designed to be extremely flexible • Most commonly used to index data in Elasticsearch

Slide 61

Slide 61 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Event Flow Simple Apache flow from input to Elasticsearch

Slide 62

Slide 62 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Logstash Pipeline • Inputs: • Network (TCP/UDP), File, syslog, stdin. • RabbitMQ, Redis • Twitter, IMAP, S3, gelf, collectd • Filters: • grok, date, mutate, ruby, geoip, etc. • Outputs: • Elasticsearch, MongoDB, File, S3 • PagerDuty, Nagios, Zabbix, Email • TCP, Redis, RabbitMQ, syslog • Graphite, Ganglia, StatsD, etc.

Slide 63

Slide 63 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Inputs!

Slide 64

Slide 64 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Inputs (50+) • Network (TCP/UDP) + File: most common • syslog / rsyslog: supports multiple simultaneous clients • RabbitMQ, Redis, Kafka: used in larger clusters • stdin: handy for "backﬁlling" data, or testing • Twitter: follow your brand's social media activity • Email (IMAP): so you don't need to read it all yourself! • Lumberjack: resilient, compressed, secure • Amazon S3, gelf, collectd, ganglia, sqs, varnishlog, etc. etc.

Slide 65

Slide 65 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Filters (60+) • grok: for extracting data using pattern matching • date: parse timestamps from fields, for use as "official" timestamps • mutate: rename, remove, replace, and modify fields in your events • ruby: run arbitrary Ruby code in the pipeline • geoip: determine geographical location based on IP address • csv: parse CSV data (or any pattern-separated data) • kv: parse key-value pairs in event data • And many, many, more

Slide 66

Slide 66 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Outputs!

Slide 67

Slide 67 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Outputs (75+) Outputs tend to ﬁt certain categories: • Storage: (Elasticsearch, MongoDB, S3, File, etc) • Notiﬁcation: (PagerDuty, Nagios, Zabbix, Email, etc.) • Relay: (TCP, Redis, RabbitMQ, Syslog, etc.) • Metrics: (Graphite, Ganglia, StatsD, etc.)

Slide 68

Slide 68 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Configuration input {} filter {} output {}

Slide 69

Slide 69 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Input • input {} • input { plugin { setting_1 => "value" } } • input {  plugin {  setting_1 => "value"  array_2 => ["value1","value2"]  hash_3 => { key => "value" }  # comment  }  }

Slide 70

Slide 70 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Filter • filter {} • filter { plugin { setting_1 => "value" } } • filter {  plugin {  setting_1 => "value"  array_2 => ["value1","value2"]  hash_3 => { key => "value" }  # comment  }  }

Slide 71

Slide 71 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Output • output {} • output { plugin { setting_1 => "value" } } • output {  plugin {  setting_1 => "value"  array_2 => ["value1","value2"]  hash_3 => { key => "value" }  # comment  }  }

Slide 72

Slide 72 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Kibana

Slide 73

Slide 73 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited What is Kibana? • Data Visualization tool • Runs in a browser (served locally or remote) • No programming necessary • Reads data from Elasticsearch • Multiple panel types • Save and share dashboards • Democratize your data!

Slide 74

Slide 74 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited table Drill into individual events.

Slide 75

Slide 75 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited histogram Plot data over time.

Slide 76

Slide 76 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited bettermap Events on a map

Slide 77

Slide 77 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited hits total hits for each query

Slide 78

Slide 78 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited stats count, total, min, max, mean

Slide 79

Slide 79 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited terms terms facet visualized

Slide 80

Slide 80 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Lovely!

Slide 81

Slide 81 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Kibana 4 Let’s build this.

Slide 82

Slide 82 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited ELK Use Cases

Slide 83

Slide 83 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Use Cases • Free search • Structured Search • Data Analytics • Log analysis • Event analysis • Visual Exploration via Kibana • Social Streams

Slide 84

Slide 84 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Scaling Elasticsearch

Slide 85

Slide 85 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Distributed by design Sharding is the unit of distribution in Elasticsearch A shard is a fully-functional Lucene Search Engine and contains many Lucene segments Primary Shard — All data is indexed here first Replica Shard — Copy of indexed data which serves 2 purposes: ‣ Increase high availability ‣ Increase read throughput

Slide 86

Slide 86 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited many segments

Slide 87

Slide 87 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited one shard s s many segments →

Slide 88

Slide 88 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited one shard s s many segments s s s s s s s s many shards s s →

Slide 89

Slide 89 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited one shard s s many segments one index I I s s s s s s s s many shards s s → →

Slide 90

Slide 90 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Scale-out with shards Shards are moved around in the cluster For performance, 1 index with 5 shards same as 5 indices with 1 shard Once index is created, cannot change # of shards Replicas can be added/increased any time

Slide 91

Slide 91 text

Slide 92

Slide 92 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited node4 Node/Shard Allocation Indices “a” and “b”, “a” with 2 shards and 1 replica, and “b” with 3 shards with 1 replica, on a 4 node cluster node1 a0 b1 b2 node3 b1 a0 node2 a1 b0 b0 a1 b2 Primary Replica

Slide 93

Slide 93 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited start small node_A shard_0 shard_1 shard_2

Slide 94

Slide 94 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited add more nodes node_A shard_0 shard_1 shard_2 node_B node_C

Slide 95

Slide 95 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited shards migrate node_A shard_0 shard_1 shard_2 node_B shard_1 node_C shard_2

Slide 96

Slide 96 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited rebalanced automatically node_A shard_0 node_B shard_1 node_C shard_2

Slide 97

Slide 97 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Removing nodes Replicas in action node_A shard_0 shard_1 node_B shard_1 replica_0 shard_0

Slide 98

Slide 98 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Removing nodes Replica was promoted to primary node_A shard_0 shard_1 shard_1

Slide 99

Slide 99 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Master node • Master manages cluster activity • By default all nodes are master ready • Master election automatic • If master node fails, another node elected automatically • Cluster has just one master at any time • Three master-eligible nodes recommended

Slide 100

Slide 100 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Hands-on Lab Elasticsearch | Logstash | Kibana

Slide 101

Slide 101 text

Slide 102

Slide 102 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Lab: Logstash Installing and Running Logstash

Slide 103

Slide 103 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Installation • Install Logstash • Create basic configuration file Goals:

Slide 104

Slide 104 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Step 1 • Obtain tarball • Uncompress into your directory • tar -zxf logstash/logstash-1.4.2.tar.gz • Result will be a directory: logstash-1.4.2 • Change directory to logstash-1.4.2 • cd logstash-1.4.2

Slide 105

Slide 105 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Step 2 Run the following test: $ bin/logstash -e \ 'input { stdin {} } output { stdout { codec => rubydebug } }' … Hello world! { "message" => "Hello world!", "@version" => "1", "@timestamp" => "2014-07-11T23:09:11.981Z", "host" => "oh-my" } 

Slide 106

Slide 106 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Step 3 • View log file logstash/sample1.log • View config file logstash/logstash-lab4.conf • Run the following with log entry found in log file $ bin/logstash -f ../logstash/logstash-lab4.conf  

Slide 107

Slide 107 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Step 4 Your results should look similar to this: { "message" => "22/Mar/2014:16:38:00 -0700 183.60.215.50 ", "@version" => "1", "@timestamp" => "2014-03-22T23:38:00.000Z", "host" => "oh-my", "ip" => "183.60.215.50", "msg" => "This", "geoip" => { … … } }

Slide 108

Slide 108 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited "geoip" => { "ip" => "183.60.215.50", "country_code2" => "CN", "country_code3" => "CHN", "country_name" => "China", "continent_code" => "AS", "region_name" => "30", "city_name" => "Guangzhou", "latitude" => 23.11670000000001, "longitude" => 113.25, "timezone" => "Asia/Chongqing", "real_region_name" => "Guangdong", "location" => [ [0] 113.25, [1] 23.11670000000001 ] }

Slide 109

Slide 109 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Lab: Elasticsearch Installing and Running Elasticsearch

Slide 110

Slide 110 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Configuration • Node configuration in config/elasticsearch.yml cluster.name: test_cluster discovery.zen.ping.multicast.enabled: false http.cors.enabled: true

Slide 111

Slide 111 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited ES Lab: Starting a Node • Extract Elasticsearch, Install and Execute # extract and cd into the directory % tar zxf elasticsearch/elasticsearch-1.4.4.tar.gz % cd elasticsearch-1.4.4 # install marvel from network or local file % bin/plugin -i elasticsearch/marvel/latest - or - % bin/plugin -i marvel -u file:../marvel/marvel-latest.zip # run in the foreground % bin/elasticsearch

Slide 112

Slide 112 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited ES Lab: Is it running? Check the result: curl 'localhost:9200' { "status" : 200, "name" : "Akhenaten", "cluster_name" : "elasticsearch", "version" : { "number" : "1.4.4", "build_hash" : "927caff6f05403e936c20bf4529f144f0c89fd8c", "build_timestamp" : "2014-12-16T14:11:12Z", "build_snapshot" : false, "lucene_version" : "4.10.2" }, "tagline" : "You Know, for Search" }

Slide 113

Slide 113 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited ES Lab: Apache Logs Storing Logstash processed logs in Elasticsearch

Slide 114

Slide 114 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited ES Lab: Populating Data • Run Logstash using this new configuration % cd logstash-1.4.4 % cp ../logstash/complete.conf . % bin/logstash -f complete.conf

Slide 115

Slide 115 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited ES Lab: Populating ES (cont.) • Copy/paste sample log • cat ../sample2.log 71.141.244.242 - kurt [18/May/2011:01:48:10 -0700] "GET /admin HTTP/1.1" 301 566 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3"

Slide 116

Slide 116 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited ES Lab: Populating ES (cont.) { "message" => "71.141.244.242 - kurt [18/May/ 2011:01:48:10 -0700] \"GET /admin HTTP/1.1\" 301 566 \"-\" \"Mozilla/5.0 (Windows; U; Windows NT 5.1; en- US; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3\"", "@version" => "1", "@timestamp" => "2011-05-18T08:48:10.000Z", "host" => "cadenza", "clientip" => "71.141.244.242", "ident" => "-", "auth" => "kurt", "timestamp" => "18/May/2011:01:48:10 -0700", "verb" => "GET", "request" => "/admin", …truncated… }

Slide 117

Slide 117 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited ES Lab: Simple Queries • Get a count of documents (in the entire cluster) % curl -XGET 'localhost:9200/logstash-*/_count?pretty' { "count" : 1, "_shards" : { "total" : 5, "successful" : 5, "failed" : 0 } }

Slide 118

Slide 118 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited ES Lab: Simple Queries • Perform a simple search for term “mozilla” curl -XGET 'localhost:9200/_search?q=mozilla&pretty' { "took" : 2, "timed_out" : false, "_shards" : { "total" : 5, "successful" : 5, "failed" : 0 }, "hits" : { "total" : 1, "max_score" : 0.047945753, "hits" : [ { "_index" : "logstash-2013.12.11", "_type" : "logs", "_id" : “z8V_NXAHQkigh-SaFW26yg", ... } ] }

Slide 119

Slide 119 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited ES Lab: Complex Search Example • Index some more documents using Logstash • Sample dataset in logs.gz file (copy into the current directory and unzip it) • Use the same configuration as previous example % gzip -d ../logs.gz % cp ../logs ./logs  % cat logs | bin/logstash -f complete.conf

Slide 120

Slide 120 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited ES Lab: Monitoring with Marvel • Simply browse to the Marvel installation! http://localhost:9200/_plugin/marvel

Slide 121

Slide 121 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited ES Lab: Aggregations • What are the top IP addresses in our apache logs? • http://bit.ly/1h2tmqt<- JSON can be found here! GET logstash-*/_search { "aggs" : { "top_uris" : { "terms" : { "field" : "clientip", "size" : 3 } } } }

Slide 122

Slide 122 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited ES Lab: Searching Elasticsearch • Results of this query (top IPs in the log) { "hits" : {… list of hits …} "aggregations" : { "top_uris" : { "buckets" : [ { "key" : "128.30.28.58", "doc_count" : 104 }, { "key" : "65.115.35.83", "doc_count" : 71 }, { "key" : "151.250.94.199", "doc_count" : 52 } … } } }

Slide 123

Slide 123 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited ES Lab: Aggregations • Exclude a specific term from the aggregation GET logstash-*/_search { "size": 0, "aggs" : { "top_uris" : { "terms" : { "field" : "clientip", "size" : 3, "exclude" : "128.30.28.58" } } } }

Slide 124

Slide 124 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited ES Lab: Aggregations Result "aggregations" : { "top_uris" : { "buckets" : [ { "key" : "65.115.35.83", "doc_count" : 71 }, { "key" : "151.250.94.199", "doc_count" : 52 }, { "key" : "46.105.14.53", "doc_count" : 5 } ] }

Slide 125

Slide 125 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Lab: Kibana Time to explore those logs.

Slide 126

Slide 126 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited First: Start Kibana • Unzip the Kibana package • Go into the ./bin directory in your terminal cd /kibana/kibana-*/bin • Start Kibana: ./kibana ./kibana.bat on Windows

Slide 127

Slide 127 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Lab 1: Start Kibana Conﬁgure Kibana by setting up the logstash-* index pattern and setting the time picker 1 2 3 Set index name pattern & set the time-field

Slide 128

Slide 128 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Lab 2: Metric Visual Click the “Visualize” tab and create a new visualization “From a new search” Click “Visualize” Create a “Metric” 1 2

Slide 129

Slide 129 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Metric Visual Now save and name your visualization “Total Events”. This is a very basic metric Save & name visualization The visualization Configure visualization 1 2

Slide 130

Slide 130 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Lab 3: Pie Chart Create a new visualization “From a new search” and choose “Pie chart” New Visualization 1 2

Slide 131

Slide 131 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Pie Chart Create a new visualization “From a new search” and choose “Pie chart” Set count Create bucket that “splits” the pie into “slices” Select “terms” aggregation Choose the “response” field Set the size and Apply 1 2 3 4 5 6

Slide 132

Slide 132 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Pie Chart Now save and name your visualization “Response Codes”. You can hover over the pie chart to get data. There is a legend on the right.

Slide 133

Slide 133 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Lab 4: Date Histogram Create a new visualization of type “Vertical bar chart”. y-axis represents the count choose x-axis bucket select “date histogram” aggregation on the field “@timestamp” leave the “Interval” on “Auto”

Slide 134

Slide 134 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Date Histogram Select “Apply” and now we can see how many events we indexed every day. Remember to save this visualization! Name it “Events Over Time”

Slide 135

Slide 135 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Lab 5: New Dashboard Now that we have some visualizations, lets create a dashboard to view them together display your visualizations 1 select them and they will appear 2

Slide 136

Slide 136 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited New Dashboard You can drag your visualizations around and re-size them. Save your dashboard! save your dashboard 1

Slide 137

Slide 137 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Lab 6: Edit Visualization Once a visualization is created it can be edited. Lets make that histogram more interesting with a “sub aggregation”! click the pencil icon to edit the visualization 1

Slide 138

Slide 138 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Edit Visualization Add a “Sub Aggregation” with a “bucket” that “Splits Bars” over a “Terms aggregation” on the “response” field

Slide 139

Slide 139 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Edit Visualization Now we can see how many of each response code occurred on each day! If you save this with the same name the visualization will be over-written. You can rename it and now you have another visualization!

Slide 140

Slide 140 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Lab 7: Line Chart It would be useful to see a time histogram that relates to diﬀerent terms. In this case, country and bandwidth over time • Create a new visualization of type “Line chart” • Change the y-axis to a “sum of bytes” aggregation • Create a date histogram on the x-axis • Add a “split lines” “terms” sub aggregation on the field “geoip.country_name.raw” • Apply and save the visualization

Slide 141

Slide 141 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Line Chart It should look something like this

Slide 142

Slide 142 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Lab 8: Tile Map Lets create a visualization that shows where events happen on a map • Create a new visualization of type “Tile map” • Add an aggregation bucket of type “Geo Coordinates” • Select “Geohash”aggregation on field “geoip.location” • In the “view options” select “Scaled Circle Markers” • Apply and save the visualization

Slide 143

Slide 143 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Tile Map You can see a lot of events in California

Slide 144

Slide 144 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Lab 9: Filter Results Now that I have a pretty nice dashboard I can also ﬁlter the results and change the time picker standard dashboard execute a search dashboard is filtered now 1 2 3

Slide 145

Slide 145 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Click a metric Show/Hide Actions filter is created Filter Results Now that I have a pretty nice dashboard I can also ﬁlter the results and change the time picker 1 2 3

Slide 146

Slide 146 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Lab 10: Apache Logs Overview Create more visualizations to get the entire view of your logs. Look at visualization descriptions to get a hint if you’re stuck • Save your work while you build this dashboard! (Bar chart) Histogram on response codes with interval 100 (Line chart) Bytes Date Histogram (Pie chart) Doughnut chart for User agents by device by country (Data table) Top 10 IPs (Data table) Top 10 Requests (Line chart) Split chart of response codes by

Slide 147

Slide 147 text

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Resources • Support: http://www.elastic.co/support • Community Resources: • irc: #logstash and #elasticsearch on freenode • email: [email protected] • email: [email protected] • meetups: http://elasticsearch.meetup.com/ • twitter: @elastic • github: https://github.com/elastic/