A deeper look at the ELK Stack: Elasticsearch, Logstash & Kibana

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
distributing without written permission is strictly prohibited
Aaron Mildenstein (@theuntergeek)
Elastic, Inc.
A deeper look
at the ELK Stack

View Slide

Lab Resources
• https://goo.gl/EttDLR
(https://s3.amazonaws.com/elk-
workshop.elasticsearch.org/hands-on-workshop/20150509-openwest/hands-on-
workshop.tar.gz)
• USB Drive (hopefully there are enough to pass
around)

View Slide

We’ve clearly hit a nerve
13M+ cumulative downloads
Million of Downloads
0
2
4
6
8
Oct'12 Jan'13 Apr'13 Jul'13 Oct'13 Jan'14 Apr'14
Nearly 5M downloads
in the last year

View Slide

Growing Customer Base

View Slide

The History of ELK

View Slide

History of the ELK Stack
• Logstash was started in 2009 by Jordan Sissel

• Elasticsearch was ﬁrst released in 2010 by Shay Banon

• Kibana was begun in 2011 by Rashid Khan

• Elasticsearch (the company) was founded in 2012

• Rashid joined Elasticsearch in January, 2013

• Jordan joined Elasticsearch in August, 2013

• Much of the development on all three projects is now done
in-house, in addition to open source contributions

View Slide

Search Use Cases

View Slide

Unstructured search

View Slide

Structured search

View Slide

Enrichment

View Slide

Sorting

View Slide

Pagination

View Slide

Aggregation

View Slide

Suggestions

View Slide

Elasticsearch

View Slide

What is Elasticsearch?
• Document-oriented search engine

• JSON based (both document store and REST API)

• Built on top of Apache Lucene

• Schema Free / Schema-Less

• Yet enables control of schema when needed (via mappings)

• Distributed Model

• Scales Up+Out, Highly Available

• Multi-tenant data

• Dynamically create/delete indices

• API centric & RESTful

• Most functionality + cluster statistics are exposed via API

View Slide

Basic glossary
• Maecenas aliquam maecenas ligula nostra, accumsan
taciti. Sociis mauris in integer
• El eu libero cras interdum at eget habitasse
elementum est, ipsum purus pede
• Aliquet sed. Lorem ipsum dolor sit amet, ligula
suspendisse nulla pretium, rhoncus
cluster A cluster consists of one or more nodes which share the
same cluster name. Each cluster has a single master
node which is chosen automatically by the cluster and
which can be replaced automatically if the current
master node fails.
node A node is a running instance of elasticsearch which
belongs to a cluster. Multiple nodes can be started on
a single server for testing purposes, but usually you
should have one node per server.
At startup, a node will use multicast (or unicast, if
speciﬁed) to discover an existing cluster with the
same cluster name and will try to join that cluster.

View Slide

Basic glossary
index An index can be seen as a named collection of
documents. It is a logical namespace which maps to
one or more primary shards and can have zero or
more replica shards.
shard A shard is a single Apache Lucene instance. It is a low-
level “worker” unit which is managed automatically.

Shards are distributed across all nodes in the cluster, and
can move shards automatically from one node to another
in the case of node failure, or the addition of new nodes.

There are two types of shards: primary and replica.

View Slide

Basic glossary
Primary
shard
An index can have one or more primary shards
(defaults to 5) and it is not possible to change this
number after index creation. When you index a
document, it is ﬁrst indexed on the primary shard,
then on all replicas of this shard.
Replica
shard
Each primary shard can have zero or more replicas
(defaults to 1). A replica is a copy of the primary shard,
and serves two purposes:

‣ Increase high availability - a replica is another copy of the
data and will be promoted to a primary shard if the
primary fails

‣ Increase performance - get and search requests can be
handled by primary or replica shards

View Slide

Create Index API
Creating Index a with 2 shards and 1
replica (a total of 4 shards)
Creating Index b with 3 shards and 1
curl -XPUT 'localhost:9200/a' -d '{
"settings" : {
"number_of_shards" : 2,
"number_of_replicas" : 1
}
}'
curl -XPUT 'localhost:9200/b' -d '{
"settings" : {
}
}'

View Slide

curl -XPUT ‘localhost:9200/crunchbase/person/1’ -d '{
"first_name" : "Tony",
"last_name" : "Stark"
}'
Index API
target index

name
HTTP REST
method
document

JSON source
document

type
document id

View Slide

• It is possible to retrieve a speciﬁc document from the index
using its _type and _id

• The GET operation is realtime

Meaning, once a document is indexed, it is immediately available to be
retrieved using the GET API
curl -XGET 'localhost:9200/crunchbase/person/1'
Get API
target index

name
HTTP REST
operation
document type document id

View Slide

Exists API
• Check if a document is in the index

Without the overhead of loading it

• The response is based on HTTP status code

200 (OK) if exists

404 (NOT FOUND) if doesn’t exist
curl -XHEAD -i 'localhost:9200/crunchbase/person/1'

View Slide

Update API
• Update by partial data

Partial doc is merged with existing doc 
 
Non-object properties with the same key are replaced. Object properties
are recursively merged
curl -XPOST 'localhost:9200/crunchbase/person/1/_update' -d '{
"doc" : {
"first_name" : "Antonio"
}
}'

View Slide

• Deleting a speciﬁc document by _id

• Response

200 (OK) if deleted

404 if not found
{
"found" : true,
"_index" : "test",
"_type" : "person",
"_id" : "1",
"_version" : 3
}
Delete API
Indication if it

was actually found
curl -XDELETE 'localhost:9200/crunchbase/person/1'

View Slide

• Sometimes you'd like to get multiple documents in one go

Avoid round trips when using the Get API
curl 'localhost:9200/_mget' -d '{
"docs" : [
{
"_index" : "crunchbase",
"_type" : "person",
"_id" : "1"
},
{
"_index" : "marvels",
"_type" : "hero",
"_id" : "2"
"_source" : [ "first_name" ]
}
]
}'
Multi Get API
index name
document type
document id
Optionally specify

what ﬁelds should

be returned

View Slide

• Minimizes round trips when performing bulk index/delete/
update operations

• The format of a bulk request is as follows
{ "delete" : { "_index" : "crunchbase", "_type" : "person", "_id" : "2" } }\n
{ "index" : { "_index" : "crunchbase", "_type" : "person", "_id" : "1" } }\n
{ "first_name" : "Tony", "last_name" : "Stark" }\n
.
.
.
{ "create" : { "_index" : "crunchbase", "_type" : "person", "_id" : "3" } }\n
{ "first_name" : "Thor", "last_name" : "Odinson" }\n
Bulk API
each line must end
with a line break
(incl. the last line)
optional action body
action metadata

View Slide

Analysis

View Slide


View Slide

Grep!
• Which plays contain the word “darling” in the complete
works of Shakespeare?
‣ Grep it!
‣ Go over each play, word by word, and mark the play that contains it
• Linear to the number of words
• Fails at large scale

View Slide

Inverted Index
• Inverting Shakespeare
‣ Take all the plays and break them down word by word
‣ For each word, store the ids of the documents that contain it
‣ Sort all tokens (words)
• Search
First look for the relevant word (fast as words are sorted), if found,
iterate over the document ids that are associated it

View Slide


View Slide

Term Doc 1 Doc 2 Doc 3
breathe
brings
buds
but
by
can
…
damasked
darling
date
day
deaf
death
declines
delight

View Slide

Analyzers
• Analysis => Tokenization and normalization
• Analyzers => Analysis and token filters
• Token filters act on the token stream - can drop
and modify existing tokens, or add new ones.
• Out of the box, many analyzers are available —
Standard analyzer, Whitespace analyzer, language
analyzers
• Can define/build custom analyzers

View Slide

_analyze API
GET /_analyze?analyzer=whitespace&text=FOO BAR
{
"tokens": [
{
"token": "FOO",
"start_offset": 0,
"end_offset": 3,
"type": "word",
"position": 1
},
{
"token": "BAR",
"start_offset": 4,
"end_offset": 7,
"type": "word",
"position": 2
}
]

View Slide

Rich Search via Query DSL
• Queries
• Unstructured search, enables to query the data based on textual
analysis (free text search). Queries score documents by relevancy
(supports powerful custom scoring algorithms). To name a few:
• match
‣ bool (boolean)
‣ histogram
‣ Filters
• Structured search, enables narrowing the search context based on
known document structure (no scoring and very fast). To name a few:
‣ term
‣ range
‣ bool (boolean)

View Slide

Querying
Powerful and rich Query DSL
Queries are analyzed too
Near real time (from indexing to querying)
GET /_search -d '{ 
{ 
"query": { 
"match": { "tweet": "elasticsearch" } 
}
} 

View Slide

Results
{
"took": 15,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"failed": 0
},
"hits": {
"total": 1,
"max_score": 0.30685282,
"hits": [
{
"_index": "twitter",
"_type": "tweets",
"_id": "cxxV4_TST_iR2zH1GuedVQ",
"_score": 0.30685282,
"_source": {
“awesome #logstash #kibana #elasticsearch presentation with
real life use case demo by @webmat at @devopsmontreal" 

View Slide

Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
Suggestions
Look familiar?

View Slide

Analytics
Un-invert the inverted index (Field data)
Load the field data to memory
Group By — popular terms, significant terms, ranges,
dates, geolocation
Metrics — count, min, max, sum, avg, percentiles,
cardinality,
Nested aggregations helps slice and dice data

View Slide

Tweets per month
GET /_all/tweet/_search -d ' 
{ 
"aggs": { 
"tweets_by_month": { 
"date_histogram": { 
"field": "date", 
"interval": "month" 
} 
} 
} 
} 

View Slide

Aggregations
Facets
Free text search

View Slide

Aggregations
Use aggregations to build analytics tools & dashboards

View Slide

Aggregations: Buckets & Metrics
• Two categories of aggregations - Buckets & Metrics

• Buckets

Aggregations that build buckets. Each bucket is associated with some
criteria over documents. During query execution, each document is
evaluated against the created buckets and each bucket keeps track of
what documents “fall” in it. Each bucket effectively defines a set of
documents derived from the document set within the aggregations
scope.

• Metrics

Aggregations that given a set of documents, produce a single/multiple
scalar/s. Typically, metrics aggregations generate numeric stats that are
computed over a specific document set

View Slide

Bucket - terms
"aggregations": {
"states": {
"buckets": [
{
"key": "ma",
"doc_count": 841
},
{
"key": "ca",
"doc_count": 631
},
...
"key": "ny",
"doc_count": 630
},
{
"key": "nj",
"doc_count": 560
},
{
"key": "wa",
"doc_count": 525
}
]
}
}
Response
wa

16%
nj

18%
ny

20%
ca

20%
ma

26%

View Slide

Bucket - *_range
• date_range

A dedicated range aggregations that works on date fields. Ranges can
be defined as date math expressions

• ip_range

A dedicated range aggregation that works on ip fields. Ranges can be
defined as ipv4 strings

or CIDR masks
{ "from" : "now-1M", "to" : "now" }
{ "from" : "10.0.0.0", "to" : "10.0.0.128" }
{ "mask" : "10.0.0.0/25" }

View Slide

Bucket - histogram
"aggregations": {
"grades_distribution": {
"buckets": [
{
"key": 60,
"doc_count": 467
},
{
"key": 70,
"doc_count": 873
},
{
"key": 80,
"doc_count": 930
},
{
"key": 90,
"doc_count": 915
}
]
}
}
Response
0
250
500
750
1000
60 70 80 90
915
930
873
467
By default, only non-empty buckets will be returned

View Slide

Metrics - extended_stats
The following computes statistics on student exam scores over a set of
document (each representing an exam result)
{
"aggs" : {
"grades" : {
"extended_stats" : {
"field" : "grade"
}
}
}
}
"aggregations": {
"grades": {
"count": 4375,
"min": 65,
"max": 99,
"avg": 82.14765714285714,
"sum": 359396,
"sum_of_squares": 29970052,
"variance": 102.06002593959144,
"std_deviation": 10.102476228113158
}
}
Request
Response
{
"subject": "Mathematics",
"state": "CA",
"age": 8,
"grade": 69,
"male": true
}
Sample document

View Slide

Aggregations
• Enables slicing & dicing the data
• Provides multi-dimensional grouping of results. e.g. Top URLs
by country.
• Many types available
• All operate over values extracted from the documents - usually
from specific fields of the documents, but highly customizable
using scripts
‣ terms
‣ range / date_range / ip_range
‣ geo_distance / geohash_grid
‣ histogram / date_histogram
‣ stats / avg / max / min / sum / percentiles

View Slide

Our Logs Have Issues

View Slide

Problem 1: No Consistency
• Every application and device logs in its own special
way.
• Expert in each log format required to use the logs.
• Difficult to search across because of this formatting
problem.

View Slide

No Consistency
120707 0:40:34 4 Connect [email protected] on
4 Query select @@version_comment limit 1
120707 0:40:45 4 Query select * from mysql.user
120707 0:41:18 5 Query hello world

View Slide

No Consistency
120707 0:37:09 [Note] Plugin 'FEDERATED' is disabled.
120707 0:37:09 InnoDB: The InnoDB memory heap is disabled
120707 0:37:09 InnoDB: Mutexes and rw_locks use GCC atomic builtins
120707 0:37:09 InnoDB: Compressed tables use zlib 1.2.5
120707 0:37:09 InnoDB: Using Linux native AIO
120707 0:37:09 InnoDB: Initializing buffer pool, size = 128.0M
120707 0:37:09 InnoDB: Completed initialization of buffer pool

View Slide

No Consistency
# [email protected]: biz_1[biz_1] @ localhost []
# Query_time: 0.000273 Lock_time: 0.000104 Rows_sent: 1 Rows_examined: 1
SET timestamp=1255345490;
SELECT * FROM organization_details;

View Slide

No Consistency
Mar 23 22:05:24 Macintosh com.apple.launchd[1] (httpd):
Throttling respawn: Will start in 10 seconds

View Slide

Problem 2: Time Formats
130460505
Oct 11 20:21:47
[29/Apr/2011:07:05:26 +0000]
020805 13:51:24
@4000000037c219bf2ef02e94

View Slide

Problem 3: Decentralized
• Logs are spread across all of your servers
• Many servers have many different kinds of logs
• ssh + grep aren’t scalable

View Slide

Problem 4: Experts Required
• People interested in the logs often…
• Do not have access to read the logs.
• Do not have expertise to understand the data.
• Do not know where the logs are.

View Slide

Logstash

View Slide

What is Logstash?
• Event processing engine, optimized for logs

• raw data in, enriched data out

• Written in Ruby, runs on JRuby

• Simple to extend, efficient to run

• Events pass through a pipeline

• Inputs: receive data from files, network, etc.

• Filters: enrich, massage, process the event data

• Outputs: send event data to other systems

• Designed to be extremely flexible

• Most commonly used to index data in Elasticsearch

View Slide

Event Flow Simple Apache flow from input to
Elasticsearch

View Slide

Logstash Pipeline
• Inputs:

• Network (TCP/UDP), File, syslog, stdin.

• RabbitMQ, Redis

• Twitter, IMAP, S3, gelf, collectd

• Filters:

• grok, date, mutate, ruby, geoip, etc.

• Outputs:

• Elasticsearch, MongoDB, File, S3

• PagerDuty, Nagios, Zabbix, Email

• TCP, Redis, RabbitMQ, syslog

• Graphite, Ganglia, StatsD, etc.

View Slide

Inputs!

View Slide

Inputs (50+)
• Network (TCP/UDP) + File: most common

• syslog / rsyslog: supports multiple simultaneous clients

• RabbitMQ, Redis, Kafka: used in larger clusters

• stdin: handy for "backﬁlling" data, or testing

• Twitter: follow your brand's social media activity

• Email (IMAP): so you don't need to read it all yourself!

• Lumberjack: resilient, compressed, secure

• Amazon S3, gelf, collectd, ganglia, sqs, varnishlog, etc. etc.

View Slide

Filters (60+)
• grok: for extracting data using pattern matching

• date: parse timestamps from fields, for use as "official" timestamps

• mutate: rename, remove, replace, and modify fields in your events

• ruby: run arbitrary Ruby code in the pipeline

• geoip: determine geographical location based on IP address

• csv: parse CSV data (or any pattern-separated data)

• kv: parse key-value pairs in event data

• And many, many, more

View Slide

Outputs!

View Slide

Outputs (75+)
Outputs tend to ﬁt certain categories:

• Storage: (Elasticsearch, MongoDB, S3, File, etc)

• Notiﬁcation: (PagerDuty, Nagios, Zabbix, Email, etc.)

• Relay: (TCP, Redis, RabbitMQ, Syslog, etc.)

• Metrics: (Graphite, Ganglia, StatsD, etc.)

View Slide

Configuration
input {}
filter {}
output {}

View Slide

Input
• input {}
• input { plugin { setting_1 => "value" } }
• input { 
plugin { 
setting_1 => "value" 
array_2 => ["value1","value2"] 
hash_3 => { key => "value" } 
# comment 
} 
}

View Slide

Filter
• filter {}
• filter { plugin { setting_1 => "value" } }
• filter { 
plugin { 
# comment 
} 
}

View Slide

Output
• output {}
• output { plugin { setting_1 => "value" } }
• output { 
plugin { 
# comment 
} 
}

View Slide

Kibana

View Slide

What is Kibana?
• Data Visualization tool

• Runs in a browser (served locally or remote)

• No programming necessary

• Reads data from Elasticsearch

• Multiple panel types

• Save and share dashboards

• Democratize your data!

View Slide

table Drill into individual events.

View Slide

histogram Plot data over time.

View Slide

bettermap Events on a map

View Slide

hits total hits for each query

View Slide

stats count, total, min, max, mean

View Slide

terms terms facet visualized

View Slide

Lovely!

View Slide

Kibana 4
Let’s build this.

View Slide

ELK Use Cases

View Slide

Use Cases
• Free search

• Structured Search

• Data Analytics

• Log analysis

• Event analysis

• Visual Exploration via Kibana

• Social Streams

View Slide

Scaling Elasticsearch

View Slide

Distributed by design
Sharding is the unit of distribution in Elasticsearch
A shard is a fully-functional Lucene Search Engine
and contains many Lucene segments
Primary Shard — All data is indexed here first
Replica Shard — Copy of indexed data which serves
2 purposes:
‣ Increase high availability
‣ Increase read throughput

View Slide

many segments

View Slide

one shard
s
s
many segments →

View Slide

one shard
s
s
many segments
s
s
s
s
s
s
s
s
many shards
s
s
→

View Slide

one shard
s
s
many segments
one index
I
I
s
s
s
s
s
s
s
s
many shards
s
s
→
→

View Slide

Scale-out with shards
Shards are moved around in the cluster
For performance, 1 index with 5 shards same as 5
indices with 1 shard
Once index is created, cannot change # of shards
Replicas can be added/increased any time

View Slide

Create Index API
Creating Index a with 2 shards and 1
Creating Index b with 3 shards and
1 replica (a total of 6 shards)
curl -XPUT 'localhost:9200/a' -d '{
"settings" : {
}
}'
curl -XPUT 'localhost:9200/b' -d '{
"settings" : {
}
}'

View Slide

node4
Node/Shard Allocation
Indices “a” and “b”, “a” with 2 shards and 1 replica, and “b” with 3 shards
with 1 replica, on a 4 node cluster
node1
a0 b1
b2
node3
b1 a0
node2
a1 b0
b0
a1
b2
Primary
Replica

View Slide

start small
node_A
shard_0
shard_1
shard_2

View Slide

add more nodes
node_A
shard_0
shard_1
shard_2
node_B node_C

View Slide

shards migrate
node_A
shard_0
shard_1
shard_2
node_B
shard_1
node_C
shard_2

View Slide

rebalanced automatically
node_A
shard_0
node_B
shard_1
node_C
shard_2

View Slide

Removing nodes
Replicas in action
node_A
shard_0
shard_1
node_B
shard_1
replica_0
shard_0

View Slide

Removing nodes
Replica was promoted to primary
node_A
shard_0
shard_1
shard_1

View Slide

Master node
• Master manages cluster activity
• By default all nodes are master ready
• Master election automatic
• If master node fails, another node elected
automatically
• Cluster has just one master at any time
• Three master-eligible nodes recommended

View Slide

Hands-on Lab
Elasticsearch | Logstash | Kibana

View Slide

Lab Resources
• https://goo.gl/EttDLR
(https://s3.amazonaws.com/elk-
workshop.elasticsearch.org/hands-on-workshop/20150509-openwest/hands-on-
workshop.tar.gz)
• USB Drive (hopefully there are enough to pass
around)

View Slide

Lab: Logstash
Installing and Running Logstash

View Slide

Installation
• Install Logstash
• Create basic configuration file
Goals:

View Slide

Step 1
• Obtain tarball
• Uncompress into your directory
• tar -zxf logstash/logstash-1.4.2.tar.gz
• Result will be a directory: logstash-1.4.2
• Change directory to logstash-1.4.2
• cd logstash-1.4.2

View Slide

Step 2
Run the following test:
$ bin/logstash -e \
'input { stdin {} } output { stdout { codec => rubydebug } }'
…
Hello world!
{
"message" => "Hello world!",
"@version" => "1",
"@timestamp" => "2014-07-11T23:09:11.981Z",
"host" => "oh-my"
} 

View Slide

Step 3
• View log file logstash/sample1.log
• View config file logstash/logstash-lab4.conf
• Run the following with log entry found in log file
$ bin/logstash -f ../logstash/logstash-lab4.conf
 

View Slide

Step 4
Your results should look similar to this:
{
"message" => "22/Mar/2014:16:38:00 -0700
183.60.215.50 ",
"@version" => "1",
"@timestamp" => "2014-03-22T23:38:00.000Z",
"host" => "oh-my",
"ip" => "183.60.215.50",
"msg" => "This",
"geoip" => {
… …
}
}

View Slide

"geoip" => {
"ip" => "183.60.215.50",
"country_code2" => "CN",
"country_code3" => "CHN",
"country_name" => "China",
"continent_code" => "AS",
"region_name" => "30",
"city_name" => "Guangzhou",
"latitude" => 23.11670000000001,
"longitude" => 113.25,
"timezone" => "Asia/Chongqing",
"real_region_name" => "Guangdong",
"location" => [
[0] 113.25,
[1] 23.11670000000001
]
}

View Slide

Lab: Elasticsearch
Installing and Running Elasticsearch

View Slide

Configuration
• Node configuration in config/elasticsearch.yml
cluster.name: test_cluster
discovery.zen.ping.multicast.enabled: false
http.cors.enabled: true

View Slide

ES Lab: Starting a Node
• Extract Elasticsearch, Install and Execute
# extract and cd into the directory
% tar zxf elasticsearch/elasticsearch-1.4.4.tar.gz
% cd elasticsearch-1.4.4
# install marvel from network or local file
% bin/plugin -i elasticsearch/marvel/latest
- or -
% bin/plugin -i marvel -u file:../marvel/marvel-latest.zip
# run in the foreground
% bin/elasticsearch

View Slide

ES Lab: Is it running?
Check the result:
curl 'localhost:9200'
{
"status" : 200,
"name" : "Akhenaten",
"cluster_name" : "elasticsearch",
"version" : {
"number" : "1.4.4",
"build_hash" : "927caff6f05403e936c20bf4529f144f0c89fd8c",
"build_timestamp" : "2014-12-16T14:11:12Z",
"build_snapshot" : false,
"lucene_version" : "4.10.2"
},
"tagline" : "You Know, for Search"
}

View Slide

ES Lab: Apache Logs
Storing Logstash processed logs in Elasticsearch

View Slide

ES Lab: Populating Data
• Run Logstash using this new configuration
% cd logstash-1.4.4
% cp ../logstash/complete.conf .
% bin/logstash -f complete.conf

View Slide

ES Lab: Populating ES (cont.)
• Copy/paste sample log
• cat ../sample2.log
71.141.244.242 - kurt [18/May/2011:01:48:10 -0700] "GET /admin HTTP/1.1"
301 566 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.3)
Gecko/20100401 Firefox/3.6.3"

View Slide

ES Lab: Populating ES (cont.)
{
"message" => "71.141.244.242 - kurt [18/May/
2011:01:48:10 -0700] \"GET /admin HTTP/1.1\" 301 566
\"-\" \"Mozilla/5.0 (Windows; U; Windows NT 5.1; en-
US; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3\"",
"@version" => "1",
"@timestamp" => "2011-05-18T08:48:10.000Z",
"host" => "cadenza",
"clientip" => "71.141.244.242",
"ident" => "-",
"auth" => "kurt",
"timestamp" => "18/May/2011:01:48:10 -0700",
"verb" => "GET",
"request" => "/admin",
…truncated…
}

View Slide

ES Lab: Simple Queries
• Get a count of documents (in the entire cluster)
% curl -XGET 'localhost:9200/logstash-*/_count?pretty'
{
"count" : 1,
"_shards" : {
"total" : 5,
"successful" : 5,
"failed" : 0
}
}

View Slide

ES Lab: Simple Queries
• Perform a simple search for term “mozilla”
curl -XGET 'localhost:9200/_search?q=mozilla&pretty'
{
"took" : 2,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"failed" : 0
},
"hits" : {
"total" : 1,
"max_score" : 0.047945753,
"hits" : [ {
"_index" : "logstash-2013.12.11",
"_type" : "logs",
"_id" : “z8V_NXAHQkigh-SaFW26yg",
...
} ]
}

View Slide

ES Lab: Complex Search
Example
• Index some more documents using Logstash
• Sample dataset in logs.gz file (copy into the current
directory and unzip it)
• Use the same configuration as previous example
% gzip -d ../logs.gz
% cp ../logs ./logs 
% cat logs | bin/logstash -f complete.conf

View Slide

ES Lab: Monitoring with Marvel
• Simply browse to the Marvel installation!
http://localhost:9200/_plugin/marvel

View Slide

ES Lab: Aggregations
• What are the top IP addresses in our apache logs?
• http://bit.ly/1h2tmqt<- JSON can be found here!
GET logstash-*/_search
{
"aggs" : {
"top_uris" : {
"terms" : {
"field" : "clientip",
"size" : 3
}
}
}
}

View Slide

ES Lab: Searching Elasticsearch
• Results of this query (top IPs in the log)
{
"hits" : {… list of hits …}
"aggregations" : {
"top_uris" : {
"buckets" : [ {
"key" : "128.30.28.58",
"doc_count" : 104
}, {
"key" : "65.115.35.83",
"doc_count" : 71
}, {
"key" : "151.250.94.199",
"doc_count" : 52
}
…
}
}
}

View Slide

ES Lab: Aggregations
• Exclude a specific term from the aggregation
GET logstash-*/_search
{
"size": 0,
"aggs" : {
"top_uris" : {
"terms" : {
"field" : "clientip",
"size" : 3,
"exclude" : "128.30.28.58"
}
}
}
}

View Slide

ES Lab: Aggregations Result
"aggregations" : {
"top_uris" : {
"buckets" : [ {
"key" : "65.115.35.83",
"doc_count" : 71
}, {
"key" : "151.250.94.199",
"doc_count" : 52
}, {
"key" : "46.105.14.53",
"doc_count" : 5
} ]
}

View Slide

Lab: Kibana
Time to explore those logs.

View Slide

First: Start Kibana
• Unzip the Kibana package

• Go into the ./bin directory in your terminal

cd /kibana/kibana-*/bin

• Start Kibana:

./kibana

./kibana.bat on Windows

View Slide

Lab 1: Start Kibana
Conﬁgure Kibana by setting up the logstash-* index pattern and setting
the time picker
1
2
3
Set index name pattern
& set the time-field

View Slide

Lab 2: Metric Visual
Click the “Visualize” tab and create a new visualization “From a
new search”
Click “Visualize”
Create a “Metric”
1
2

View Slide

Metric Visual
Now save and name your visualization “Total Events”. This is a very basic
metric
Save & name visualization
The visualization
Configure visualization
1
2

View Slide

Lab 3: Pie Chart
Create a new visualization “From a new search” and choose “Pie
chart”
New Visualization
1
2

View Slide

Pie Chart
Create a new visualization “From a new search” and choose “Pie
chart”
Set count
Create bucket that
“splits” the pie
into “slices”
Select “terms”
aggregation
Choose the
“response”
field
Set the
size and
Apply
1
2
3
4
5
6

View Slide

Pie Chart
Now save and name your visualization “Response Codes”. You can hover
over the pie chart to get data. There is a legend on the right.

View Slide

Lab 4: Date Histogram
Create a new visualization of type “Vertical bar chart”.
y-axis represents the count
choose x-axis bucket
select “date histogram”
aggregation
on the field “@timestamp”
leave the “Interval” on “Auto”

View Slide

Date Histogram
Select “Apply” and now we can see how many events we indexed every
day. Remember to save this visualization! Name it “Events Over Time”

View Slide

Lab 5: New Dashboard
Now that we have some visualizations, lets create a dashboard to view
them together
display your visualizations 1
select them and they will appear
2

View Slide

New Dashboard
You can drag your visualizations around and re-size them. Save your
dashboard!
save your
dashboard
1

View Slide

Lab 6: Edit Visualization
Once a visualization is created it can be edited. Lets make that
histogram more interesting with a “sub aggregation”!
click the pencil icon
to edit the
visualization
1

View Slide

Edit Visualization
Add a “Sub Aggregation” with a “bucket” that “Splits Bars” over a
“Terms aggregation” on the “response” field

View Slide

Edit Visualization
Now we can see how many of each response code occurred on each day! If you
save this with the same name the visualization will be over-written. You can
rename it and now you have another visualization!

View Slide

Lab 7: Line Chart
It would be useful to see a time histogram that relates to diﬀerent terms.
In this case, country and bandwidth over time
• Create a new visualization of type “Line chart”
• Change the y-axis to a “sum of bytes” aggregation
• Create a date histogram on the x-axis
• Add a “split lines” “terms” sub aggregation on the field
“geoip.country_name.raw”
• Apply and save the visualization

View Slide

Line Chart
It should look something like this

View Slide

Lab 8: Tile Map
Lets create a visualization that shows where events happen on a map
• Create a new visualization of type “Tile map”
• Add an aggregation bucket of type “Geo Coordinates”
• Select “Geohash”aggregation on field “geoip.location”
• In the “view options” select “Scaled Circle Markers”
• Apply and save the visualization

View Slide

Tile Map
You can see a lot of events in California

View Slide

Lab 9: Filter Results
Now that I have a pretty nice dashboard I can also ﬁlter the results and
change the time picker
standard
dashboard
execute a search
dashboard is
filtered now
1
2
3

View Slide

Click a metric
Show/Hide Actions
filter is created
Filter Results
Now that I have a pretty nice dashboard I can also ﬁlter the results and
change the time picker
1
2
3

View Slide

Lab 10: Apache Logs Overview
Create more visualizations to get the entire view of your logs. Look at
visualization descriptions to get a hint if you’re stuck
• Save your work while you build this dashboard!
(Bar chart)
Histogram on
response codes with
interval 100
(Line chart)
Bytes
Date Histogram
(Pie chart)
Doughnut chart for
User agents by
device by country
(Data table)
Top 10 IPs
(Data table)
Top 10
Requests
(Line chart)
Split chart of
response
codes by

View Slide

Resources
• Support: http://www.elastic.co/support
• Community Resources:
• irc: #logstash and #elasticsearch on freenode
• email: [email protected]
• email: [email protected]
• meetups: http://elasticsearch.meetup.com/
• twitter: @elastic
• github: https://github.com/elastic/

View Slide

A deeper look at the ELK Stack: Elasticsearch, Logstash & Kibana

A deeper look at the ELK Stack: Elasticsearch, Logstash & Kibana

Aaron Mildenstein

More Decks by Aaron Mildenstein

Other Decks in Technology

Featured

Transcript