Slide 1
Slide 1 text
Logs
&
metrics
Use the force, gain the insight
By Thijs Feryn
Slide 2
Slide 2 text
Hi, I’m Thijs
Slide 3
Slide 3 text
I’m
@ThijsFeryn
on Twitter
Slide 4
Slide 4 text
I’m an
at
Evangelist
Slide 5
Slide 5 text
I’m a
at
board member
Slide 6
Slide 6 text
No content
Slide 7
Slide 7 text
Insight
&
Knowledge
Slide 8
Slide 8 text
Insight & knowledge
✓Google Analytics
✓Server logs
✓Application logs
✓Server resource usage
✓Server alerts
Slide 9
Slide 9 text
Strategy
Slide 10
Slide 10 text
Hold on to your butts
Slide 11
Slide 11 text
Chapter 1
Centralizing logs
Slide 12
Slide 12 text
Traditional log setup
Server 1 Server 2 Server 3
• access_log
• error_log
• syslog
• access_log
• error_log
• syslog
• access_log
• error_log
• syslog
Slide 13
Slide 13 text
Search?
✓Grep
✓Separate search on each machine
✓No graphing
✓Resource intensive on large files
Slide 14
Slide 14 text
syslog-ng
Slide 15
Slide 15 text
syslog-ng?
✓Still grep
✓All logs are centrally stored
✓Still separate files per server per date
✓Still no graphing
✓Still resource intensive on large files
Slide 16
Slide 16 text
What do we need?
Slide 17
Slide 17 text
What do we need?
✓Single storage point
✓Search all data at once
✓Fast & scalable search
✓Filter data
✓Convert logs into usable data
✓Visualize data
Slide 18
Slide 18 text
Logstash can do that!
http://logstash.net
Slide 19
Slide 19 text
Input Output
Filter
Files Logstash daemon Files/database/
service
Slide 20
Slide 20 text
collectd
drupal_dblog
elasticsearch
eventlog
exec
file
ganglia
gelf
gemfire
generator
graphite
heroku
imap
invalid_input
irc
jmx
log4j
lumberjack
pipe
puppet_facter
rabbitmq
rackspace
redis
relp
s3
snmptrap
sqlite
sqs
stdin
stomp
syslog
tcp
twitter
udp
unix
varnishlog
websocket
wmi
xmpp
zenoss
zeromq
Logstash input
Slide 21
Slide 21 text
advisor
alter
anonymize
checksum
cidr
cipher
clone
collate
csv
date
dns
drop
elapsed
elasticsearch
environment
extractnumbers
fingerprint
gelfify
geoip
grep
grok
grokdiscovery
i18n
json
json_encode
kv
metaevent
metrics
multiline
mutate
noop
prune
punct
railsparallelrequest
range
ruby
sleep
split
sumnumbers
syslog_pri
throttle
translate
unique
urldecode
useragent
uuid
wms
wmts
xml
zeromq
Logstash filter
Slide 22
Slide 22 text
boundary
circonus
cloudwatch
csv
datadog
datadog_metrics
elasticsearch
elasticsearch_http
elasticsearch_river
email
exec
file
ganglia
gelf
gemfire
google_bigquery
google_cloud_storage
graphite
graphtastic
hipchat
http
irc
jira
juggernaut
librato
loggly
lumberjack
metriccatcher
mongodb
nagios
nagios_nsca
null
opentsdb
pagerduty
pipe
rabbitmq
rackspace
redis
redmine
riak
riemann
s3
sns
solr_http
sqs
statsd
stdout
stomp
syslog
tcp
udp
websocket
xmpp
zabbix
zeromq
Logstash output
Slide 23
Slide 23 text
Let’s try
something simple
Slide 24
Slide 24 text
217.21.177.69
-‐
-‐
[03/Jun/2014:12:37:52
+0000]
"GET
/
HTTP/1.1"
200
483
"-‐"
"Mozilla/5.0
(Macintosh;
Intel
Mac
OS
X
10.9;
rv:29.0)
Gecko/20100101
Firefox/29.0"
/var/log/apache2/
access.log
Slide 25
Slide 25 text
input
{
file
{
type
=>
"apache"
path
=>
"/var/log/apache2/access.log"
}
}
output
{
stdout
{
codec
=>
rubydebug
}
}
/etc/logstash/conf.d/logtash.conf
Slide 26
Slide 26 text
{
"tags"
=>
[],
"@version"
=>
1,
"@timestamp"
=>
"2014-‐06-‐03T13:13:10.036+00:00",
"host"
=>
"meten",
"file"
=>
"/var/log/apache2/access.log",
"message"
=>
"217.21.177.69
-‐
-‐
[03/Jun/
2014:13:13:10
+0000]
\"GET
/
HTTP/1.1\"
200
483
\"-‐\"
\"Mozilla/5.0
(Macintosh;
Intel
Mac
OS
X
10.9;
rv:
29.0)
Gecko/20100101
Firefox/29.0\"",
"type"
=>
"apache"
}
Nothing
special, right?
Slide 27
Slide 27 text
Let’s apply a filter
Slide 28
Slide 28 text
input
{
file
{
type
=>
"apache"
path
=>
"/var/log/apache2/access.log"
}
}
filter
{
grok
{
type
=>
"apache"
match
=>
["message","%{COMBINEDAPACHELOG}"]
}
}
output
{
stdout
{
codec
=>
rubydebug
}
}
Slide 29
Slide 29 text
{
"tags"
=>
[],
"@version"
=>
1,
"@timestamp"
=>
"2014-‐06-‐03T12:58:19.411+00:00",
"host"
=>
"meten",
"file"
=>
"/var/log/apache2/access.log",
"message"
=>
"217.21.177.69
-‐
-‐
[03/Jun/2014:12:58:19
+0000]
\"GET
/
HTTP/1.1\"
200
482
\"-‐\"
\"Mozilla/5.0
(Macintosh;
Intel
Mac
OS
X
10.9;
rv:29.0)
Gecko/20100101
Firefox/29.0\"",
"type"
=>
"apache",
"clientip"
=>
"217.21.177.69",
"ident"
=>
"-‐",
"auth"
=>
"-‐",
"timestamp"
=>
"03/Jun/2014:12:58:19
+0000",
"verb"
=>
"GET",
"request"
=>
"/",
"httpversion"
=>
"1.1",
"response"
=>
"200",
"bytes"
=>
"482",
"referrer"
=>
"\"-‐\"",
"agent"
=>
"\"Mozilla/5.0
(Macintosh;
Intel
Mac
OS
X
10.9;
rv:29.0)
Gecko/20100101
Firefox/29.0\""
}
Looks more like
it, huh?
Slide 30
Slide 30 text
input
{
file
{
type
=>
"apache"
path
=>
"/var/log/apache2/access.log"
}
}
filter
{
grok
{
type
=>
"apache"
match
=>
["message","%{COMBINEDAPACHELOG}"]
}
}
output
{
stdout
{
codec
=>
rubydebug
}
}
What’s this
GROK thing?
Slide 31
Slide 31 text
Grok is currently the best way in logstash
to parse crappy unstructured log data into
something structured and queryable.
This tool is perfect for syslog logs, apache
and other webserver logs, mysql logs, and
in general, any log format that is generally
written for humans and not computer
consumption.
Logstash ships with about 120 patterns by
default. You can find them here: https://
github.com/logstash/logstash/tree/v1.4.1/
patterns. You can add your own trivially.
(See the patterns_dir setting)
If you need help building patterns to match
your logs, you will find the http://
grokdebug.herokuapp.com too quite
useful!
Slide 32
Slide 32 text
Grok patterns
123 Thijs Feryn
%{number:myNumber}
%{firstname:myFirstname}
%
{lastname:myLastname}
number
[0-‐9+]
firstname
[A-‐Z][a-‐z]+
lastname
%{firstname}
Input
Pattern
Custom
patterns
Slide 33
Slide 33 text
{
"myNumber":
[
[
"3"
]
],
"myFirstname":
[
[
"Thijs"
]
],
"myLastname":
[
[
"Feryn"
]
],
"firstname":
[
[
"Feryn"
]
]
}
Grok patterns
Result
Slide 34
Slide 34 text
SYSLOGTIMESTAMP
%{MONTH}
+%{MONTHDAY}
%{TIME}
PROG
(?:[\w._/%-‐]+)
SYSLOGPROG
%{PROG:program}(?:\[%{POSINT:pid}\])?
SYSLOGHOST
%{IPORHOST}
SYSLOGFACILITY
<%{NONNEGINT:facility}.%{NONNEGINT:priority}
>
HTTPDATE
%{MONTHDAY}/%{MONTH}/%{YEAR}:%{TIME}
%{INT}
!
!
QS
%{QUOTEDSTRING}
!
SYSLOGBASE
%{SYSLOGTIMESTAMP:timestamp}
(?:%
{SYSLOGFACILITY}
)?%{SYSLOGHOST:logsource}
%{SYSLOGPROG}:
COMMONAPACHELOG
%{IPORHOST:clientip}
%{USER:ident}
%
{USER:auth}
\[%{HTTPDATE:timestamp}\]
"(?:%{WORD:verb}
%
{NOTSPACE:request}(?:
HTTP/%{NUMBER:httpversion})?|%
{DATA:rawrequest})"
%{NUMBER:response}
(?:%
{NUMBER:bytes}|-‐)
COMBINEDAPACHELOG
%{COMMONAPACHELOG}
%{QS:referrer}
%
{QS:agent}
Slide 35
Slide 35 text
Let’s add another
cool filter
Slide 36
Slide 36 text
input
{
file
{
type
=>
"apache"
path
=>
"/var/log/apache2/access.log"
}
}
filter
{
grok
{
type
=>
"apache"
match
=>
["message","%{COMBINEDAPACHELOG}"]
}
geoip
{
source
=>
"clientip"
target
=>
"geoip"
}
}
output
{
stdout
{
codec
=>
rubydebug
}
}
Slide 37
Slide 37 text
{
"tags"
=>
[],
"@version"
=>
1,
"@timestamp"
=>
"2014-‐06-‐03T13:57:04.498+00:00",
"host"
=>
"meten",
"file"
=>
"/var/log/apache2/access.log",
"message"
=>
"217.21.177.69
-‐
-‐
[03/Jun/2014:13:33:22
+0000]
\"GET
/
HTTP/1.1\"
200
482
\"-‐\"
\"Mozilla/5.0
(Macintosh;
Intel
Mac
OS
X
10.9;
rv:29.0)
Gecko/20100101
Firefox/
29.0\"",
"type"
=>
"apache",
"clientip"
=>
"217.21.177.69",
"ident"
=>
"-‐",
"auth"
=>
"-‐",
"timestamp"
=>
"03/Jun/2014:13:33:22
+0000",
"verb"
=>
"GET",
"request"
=>
"/",
"httpversion"
=>
"1.1",
"response"
=>
"200",
"bytes"
=>
"482",
"referrer"
=>
"\"-‐\"",
"agent"
=>
"\"Mozilla/5.0
(Macintosh;
Intel
Mac
OS
X
10.9;
rv:29.0)
Gecko/20100101
Firefox/29.0\"",
"geoip"
=>
{
"ip"
=>
"217.21.177.69",
"country_code2"
=>
"BE",
"country_code3"
=>
"BEL",
"country_name"
=>
"Belgium",
"continent_code"
=>
"EU",
"region_name"
=>
"08",
"city_name"
=>
"Gent",
"latitude"
=>
51.05000000000001,
"longitude"
=>
3.75,
"timezone"
=>
"Europe/Brussels",
"real_region_name"
=>
"Oost-‐Vlaanderen",
"location"
=>
[
[0]
3.75,
[1]
51.05000000000001
]
}
GeoIP magic
Slide 38
Slide 38 text
"geoip"
=>
{
"ip"
=>
"217.21.177.69",
"country_code2"
=>
"BE",
"country_code3"
=>
"BEL",
"country_name"
=>
"Belgium",
"continent_code"
=>
"EU",
"region_name"
=>
"08",
"city_name"
=>
"Gent",
"latitude"
=>
51.05000000000001,
"longitude"
=>
3.75,
"timezone"
=>
"Europe/Brussels",
"real_region_name"
=>
"Oost-‐Vlaanderen",
"location"
=>
[
[0]
3.75,
[1]
51.05000000000001
]
}
Slide 39
Slide 39 text
Let’s send these logs
some place useful
Slide 40
Slide 40 text
Give it up for
http://elasticsearch.org
Slide 41
Slide 41 text
Elasticsearch is a flexible and powerful open
source, distributed, real-time search and
analytics engine. Architected from the ground
up for use in distributed environments where
reliability and scalability are must haves,
Elasticsearch gives you the ability to move
easily beyond simple full-text search. Through its
robust set of APIs and query DSLs, plus clients for
the most popular programming languages,
Elasticsearch delivers on the near limitless
promises of search technology
Slide 42
Slide 42 text
input
{
file
{
type
=>
"apache"
path
=>
"/var/log/apache2/access.log"
}
}
filter
{
grok
{
type
=>
"apache"
match
=>
["message","%{COMBINEDAPACHELOG}"]
}
geoip
{
source
=>
"clientip"
target
=>
"geoip"
}
}
output
{
stdout
{
codec
=>
rubydebug
}
elasticsearch
{
host
=>
"localhost"
}
}
Slide 43
Slide 43 text
Logstash + ElasticSearch
• access_log
• error_log
• syslog
• access_log
• error_log
• syslog
• access_log
• error_log
• syslog
ElasticSearch
Server 2
+
Logstash
Server 1
+
Logstash
Server 3
+
Logstash
Slide 44
Slide 44 text
Logstash ElasticSearch glossary
✓Database is called an „index”
✓Table is called a „type”
✓Row is called a „document”
✓Schema-less (sort of), but Logstash
applies its own custom schema
✓Logstash creates a daily index
✓Input types are stored in ElasticSearch
types
✓Each log is a document of a certain type
in the index of that day.
Slide 45
Slide 45 text
Querying
ElasticSearch
curl
http://localhost:9200/_search?pretty
Slide 46
Slide 46 text
{
"took"
:
36,
"timed_out"
:
false,
"_shards"
:
{
"total"
:
5,
"successful"
:
5,
"failed"
:
0
},
"hits"
:
{
"total"
:
1,
"max_score"
:
1.0,
"hits"
:
[
{
"_index"
:
"logstash-‐2014.06.03",
"_type"
:
"apache",
"_id"
:
"OS81NlYWTQu4j4O5OZMmXA",
"_score"
:
1.0,
"_source":{"tags":[],"@version":
1,"@timestamp":"2014-‐06-‐03T14:37:07.119Z","host":"meten","file":"/var/log/apache2/
access.log","message":"10.10.10.1
-‐
-‐
[03/Jun/2014:14:37:07
+0000]
\"GET
/
HTTP/1.1\"
200
483
\"-‐\"
\"Mozilla/5.0
(Macintosh;
Intel
Mac
OS
X
10.9;
rv:29.0)
Gecko/20100101
Firefox/
29.0\"","type":"apache","clientip":"217.21.177.69","ident":"-‐","auth":"-‐","timestamp":"03/
Jun/2014:14:37:07
+0000","verb":"GET","request":"/","httpversion":"1.1","response":"200","bytes":"483","referr
er":"\"-‐\"","agent":"\"Mozilla/5.0
(Macintosh;
Intel
Mac
OS
X
10.9;
rv:29.0)
Gecko/20100101
Firefox/29.0\"","geoip":
{"ip":"217.21.177.69","country_code2":"BE","country_code3":"BEL","country_name":"Belgium","c
ontinent_code":"EU","region_name":"08","city_name":"Gentbrugge","latitude":
51.05000000000001,"longitude":3.75,"timezone":"Europe/Brussels","real_region_name":"Oost-‐
Vlaanderen","location":[3.75,51.05000000000001]}}
}
]
}
}
Slide 47
Slide 47 text
"hits"
:
{
"total"
:
1,
"max_score"
:
1.0,
"hits"
:
[
{
"_index"
:
"logstash-‐2014.06.03",
"_type"
:
"apache",
"_id"
:
"OS81NlYWTQu4j4O5OZMmXA",
"_score"
:
1.0,
"_source":{"tags":[],"@version":
1,"@timestamp":"2014-‐06-‐03T14:37:07.119Z","host":"meten","file":
access.log","message":"10.10.10.1
-‐
-‐
[03/Jun/2014:14:37:07
+000
\"-‐\"
\"Mozilla/5.0
(Macintosh;
Intel
Mac
OS
X
10.9;
rv:29.0)
Ge
29.0\"","type":"apache","clientip":"217.21.177.69","ident":"-‐","
2014:14:37:07
+0000","verb":"GET","request":"/","httpversion":"1.1","response"
"\"-‐\"","agent":"\"Mozilla/5.0
(Macintosh;
Intel
Mac
OS
X
10.9;
29.0\"","geoip":
{"ip":"217.21.177.69","country_code2":"BE","country_code3":"BEL"
nent_code":"EU","region_name":"08","city_name":"Gentbrugge","lat
51.05000000000001,"longitude":3.75,"timezone":"Europe/Brussels",
Vlaanderen","location":[3.75,51.05000000000001]}}
}
]
Slide 48
Slide 48 text
{
"template"
:
"logstash-‐*",
"settings"
:
{
"index.refresh_interval"
:
"5s"
},
"mappings"
:
{
"_default_"
:
{
"_all"
:
{"enabled"
:
true},
"dynamic_templates"
:
[
{
"string_fields"
:
{
"match"
:
"*",
"match_mapping_type"
:
"string",
"mapping"
:
{
"type"
:
"string",
"index"
:
"analyzed",
"omit_norms"
:
true,
"fields"
:
{
"raw"
:
{"type":
"string",
"index"
:
"not_analyzed",
"ignore_above"
:
256}
}
}
}
}
],
"properties"
:
{
"@version":
{
"type":
"string",
"index":
"not_analyzed"
},
"geoip"
:
{
"type"
:
"object",
"dynamic":
true,
"path":
"full",
"properties"
:
{
"location"
:
{
"type"
:
"geo_point"
}
}
}
}
}
}
}
Logstash
mapping
Slide 49
Slide 49 text
How is that useful?
Slide 50
Slide 50 text
Logstash ElasticSearch
✓Centralized storage
✓„Infinitely” scalable ( -> clustering)
✓Fast & detailed search
✓Automatic schema mapping
✓Simple to configure
✓Simple to query
✓Ménage à 3
Slide 51
Slide 51 text
http://www.elasticsearch.org/overview/kibana/
Slide 52
Slide 52 text
Kibana
✓Reads Logstash data out of ElasticSearch
✓Written entirely in HTML, Javascript & CSS
(no server-side code)
✓Visualizes logs
✓Customizable panels
✓Supported by the ElasticSearch &
Logstash team
✓Almost zero config
Slide 53
Slide 53 text
Logstash
!
ElasticSearch
!
Kibana
Slide 54
Slide 54 text
Logstash
!
ElasticSearch
!
Kibana
The
ELK
stack
Slide 55
Slide 55 text
Out of the
box
Slide 56
Slide 56 text
No content
Slide 57
Slide 57 text
Search
Visualize
Overview
Filter
Slide 58
Slide 58 text
Let’s use the
location data
Slide 59
Slide 59 text
No content
Slide 60
Slide 60 text
No content
Slide 61
Slide 61 text
No content
Slide 62
Slide 62 text
No content
Slide 63
Slide 63 text
Do it yourself
Slide 64
Slide 64 text
curl
http://weten.dev:9200/_search?pretty
-‐d'
{
"from":
0,
"size":
100,
"query"
:
{
"filtered"
:
{
"query"
:
{
"match_all"
:
{}
},
"filter"
:
{
"and"
:
[
{
"range":
{
"@timestamp"
:
{
"from":
"2014-‐06-‐02T10:00:00Z",
"to":
"2014-‐06-‐04T16:00:00Z"
}
}
},
{
"term":
{
"geoip.country_code2.raw":
"DE"
}
}
]
}
}
},
"sort"
:
[
{"@timestamp"
:
{"order"
:
"desc"}}
]
}'
Slide 65
Slide 65 text
curl
http://weten.dev:9200/logstash-‐2014.06.04/apache/_search?pretty
-‐
d'
{
"from":
0,
"size":
100,
"query"
:
{
"filtered"
:
{
"query"
:
{
"match_all"
:
{}
},
"filter"
:
{
"term":
{
"geoip.country_code2.raw":
"DE"
}
}
}
},
"sort"
:
[
{"@timestamp"
:
{"order"
:
"desc"}}
]
}'
~database ~table
Slide 66
Slide 66 text
http://weten.dev:9200/_search?pretty
http://weten.dev:9200/logstash-‐2014.06.03/_search?
pretty
http://weten.dev:9200/logstash-‐2014.06.*/apache/
_search?pretty
http://weten.dev:9200/
logstash-‐2014.06.03,logstash-‐2014.06.04/apache/
_search?pretty
Index/type variations
Slide 67
Slide 67 text
Aggregations
group by on steroids
http://www.elasticsearch.org/guide/en/elasticsearch/
reference/current/search-aggregations.html
Slide 68
Slide 68 text
curl
"http://weten.dev:9200/_search?pretty"
-‐d'
{
"size":
0,
"aggregations":
{
"countries":
{
"terms":
{
"field":
"geoip.country_name.raw"
}
}
}
}'
Value counting
Slide 69
Slide 69 text
"total"
:
16,
"max_score"
:
0.0,
"hits"
:
[
]
},
"aggregations"
:
{
"countries"
:
{
"buckets"
:
[
{
"key"
:
"Belgium",
"doc_count"
:
8
},
{
"key"
:
"Germany",
"doc_count"
:
7
},
{
"key"
:
"United
Kingdom",
"doc_count"
:
1
}
]
}
}
}
Slide 70
Slide 70 text
curl
http://weten.dev:9200/_search?pretty
-‐d'
{
"aggs"
:
{
"how_far_from_amsterdam"
:
{
"geo_distance"
:
{
"field"
:
"geoip.location",
"origin"
:
"52.3760,
4.894",
"unit"
:
"km",
"ranges"
:
[
{
"to"
:
100
},
{
"from"
:
100,
"to"
:
300
},
{
"from"
:
300
}
]
}
}
}
}'
Geo distance
Slide 71
Slide 71 text
"hits"
:
[
]
},
"aggregations"
:
{
"how_far_from_amsterdam"
:
{
"buckets"
:
[
{
"key"
:
"*-‐100.0",
"from"
:
0.0,
"to"
:
100.0,
"doc_count"
:
0
},
{
"key"
:
"100.0-‐300.0",
"from"
:
100.0,
"to"
:
300.0,
"doc_count"
:
8
},
{
"key"
:
"300.0-‐*",
"from"
:
300.0,
"doc_count"
:
8
}
]
}
}
}
Slide 72
Slide 72 text
Should Logstash run
on all my machines?
Slide 73
Slide 73 text
Nope! That would
be kinda heavy
on your system
Slide 74
Slide 74 text
Ship logs to a
central Logstash
system
Slide 75
Slide 75 text
Centralized Logstash
Server 1 Server 2 Server 3
Redis
Logstash
ElasticSearch
Shipper
required
Slide 76
Slide 76 text
Logstash shippers
✓Beaver (python)
✓Woodchuck (ruby)
✓Awesant (perl)
✓Lumberjack (c)
✓Syslog-shipper (ruby)
✓Remote_syslog (ruby)
✓Message::Passing (perl)
✓Nxlog (c)
✓Logtail (perl)
✓Node-logstash (javascript)
Slide 77
Slide 77 text
I like Beaver
https://github.com/josegonzalez/beaver
http://redis.io
It can use Redis as a
message queue
Slide 78
Slide 78 text
[beaver]
transport:
redis
redis_url:
redis://10.10.10.37:6379/0
redis_namespace:
apache
logstash_version:
1
[/var/log/apache2/access.log]
type:
apache
Beaver
conf file
Slide 79
Slide 79 text
input
{
redis
{
host
=>
"localhost"
type
=>
"apache"
key
=>
"apache"
data_type
=>
"list"
}
}
filter
{
grok
{
type
=>
"apache"
match
=>
["message","%{COMBINEDAPACHELOG}"]
}
geoip
{
source
=>
"clientip"
target
=>
"geoip"
}
}
output
{
stdout
{
codec
=>
rubydebug
}
elasticsearch
{
host
=>
"localhost"
}
}
Logstash
conf file
Slide 80
Slide 80 text
Chapter 2
Server metrics
Slide 81
Slide 81 text
Other
measurement
points
Not
coming from
logs
Slide 82
Slide 82 text
Collectd
Collect server metrics
https://collectd.org
Slide 83
Slide 83 text
LoadPlugin
apache
LoadPlugin
cpu
LoadPlugin
df
LoadPlugin
disk
LoadPlugin
interface
LoadPlugin
load
LoadPlugin
memory
LoadPlugin
processes
LoadPlugin
swap
LoadPlugin
write_graphite
URL
"http://localhost/server-‐status?auto"
Host
"10.10.10.37"
Port
"2003"
StoreRates
false
AlwaysAppendDS
false
EscapeCharacter
"_"
/etc/collectd/
collectd.conf
What’s Graphite?
Slide 84
Slide 84 text
Graphite
Store numeric, time-series
data & render real-time
graphs on demand
http://graphite.wikidot.com
Slide 85
Slide 85 text
No content
Slide 86
Slide 86 text
Graphite is not just
a dashboard, it
comes with an API
Slide 87
Slide 87 text
http://weten.dev/render?
from=-‐1minutes&until=now&width=400&height=2
50&target=meten_dev.load.load.shortterm&are
aMode=stacked&connectedLimit=&_uniq=0.31136
49664673026&title=meten_dev.load.load.short
term
Slide 88
Slide 88 text
http://weten.dev/render?
from=-‐1minutes&until=now&width=400&height=2
50&target=meten_dev.load.load.shortterm&are
aMode=stacked&connectedLimit=&_uniq=0.31136
49664673026&title=meten_dev.load.load.short
term&format=json
Slide 89
Slide 89 text
[{"target":
"meten_dev.load.load.shortterm",
"datapoints":
[[null,
1401810412],
[null,
1401810413],
[null,
1401810414],
[1.27,
1401810415],
[null,
1401810416],
[null,
1401810417],
[null,
1401810418],
[null,
1401810419],
[null,
1401810420],
[null,
1401810421],
[null,
1401810422],
[null,
1401810423],
[null,
1401810424],
[1.07,
1401810425],
[null,
1401810426],
[null,
1401810427],
[null,
1401810428],
[null,
1401810429],
[null,
1401810430],
[null,
1401810431],
[null,
1401810432],
[null,
1401810433],
[null,
1401810434],
[1.05,
1401810435],
[null,
1401810436],
[null,
1401810437],
[null,
1401810438],
[null,
1401810439],
[null,
1401810440],
[null,
1401810441],
[null,
1401810442],
[null,
1401810443],
[null,
1401810444],
[1.59,
1401810445],
[null,
1401810446],
[null,
1401810447],
[null,
1401810448],
[null,
1401810449],
[null,
1401810450],
[null,
1401810451],
[null,
1401810452],
[null,
Slide 90
Slide 90 text
Graphite output formats
✓Raw
✓Json
✓Png
✓Csv
✓Svg
Slide 91
Slide 91 text
The Graphite
dashboard looks
kinda ugly, right?
Slide 92
Slide 92 text
No content
Slide 93
Slide 93 text
There are
alternatives that
leverage the API
Slide 94
Slide 94 text
Alternative Graphite
dashboards
✓Giraffe
✓Graphene
✓GDash
✓…
http://graphite.readthedocs.org/en/latest/tools.html
Slide 95
Slide 95 text
Let’s have a look at
Graphene
Slide 96
Slide 96 text
http://jondot.github.io/graphene/
Slide 97
Slide 97 text
Dashboard
Slide 98
Slide 98 text
description
=
{
"Load"
:
{
source:
"http://weten.dev/render?
from=-‐2minutes&until=now&target=meten_dev.load.load.s
hortterm&format=json",
TimeSeries:
{
parent:
"#graph",
title:
"Load",
ymax:
"40",
width:
"800",
height:
"300",
},
}
}
var
g
=
new
Graphene;
g.build(description);
Slide 99
Slide 99 text
It turns this …
Slide 100
Slide 100 text
No content
Slide 101
Slide 101 text
Into this …
Slide 102
Slide 102 text
Chapter 3
Application
metrics
Slide 103
Slide 103 text
Sending real-time
stats from your
within your code
Slide 104
Slide 104 text
Statsd
https://github.com/etsy/statsd/
Slide 105
Slide 105 text
Statsd
✓Written in Nodejs
✓Listens for stats, counters & timers
✓Communicates over UDP
✓Pluggable backends (e.g. Graphite)
✓Client libraries for all popular
languages
Slide 106
Slide 106 text
Statsd
node
stats.js
config.js
{
graphitePort:
2003
,
graphiteHost:
"graphite.example.com"
,
port:
8125
,
backends:
[
"./backends/graphite"
]
}
example
config
Slide 107
Slide 107 text
PHP & Statsd
https://github.com/domnikl/statsd-php
increment("foo.bar");!
$statsd->decrement("foo.bar");!
$statsd->count("foo.bar", 1000);
Slide 108
Slide 108 text
setNamespace("test");!
!
// simple counts!
$statsd->increment("foo.bar");!
$statsd->decrement("foo.bar");!
$statsd->count("foo.bar", 1000);!
!
// timing!
$statsd->timing("foo.bar", 320);!
$statsd->time("foo.bar.bla", function() {!
// code to be measured goes here ...!
});!
!
// more complex timings can be handled with startTiming() and endTiming(
)!
$statsd->startTiming("foo.bar");!
// more complex code here ...!
$statsd->endTiming("foo.bar");!
!
// memory profiling!
$statsd->startMemoryProfile('memory.foo');!
// some complex code goes here ...!
$statsd->endMemoryProfile('memory.foo');!
!
// report peak usage!
$statsd->memory('foo.memory_peak_usage');!
!
// gauges!
$statsd->gauge('foobar', 3);
PHP & Statsd
Slide 109
Slide 109 text
Statsd client libraries
✓pystatsd (https://github.com/sivy/pystatsd)
✓statsd-ruby (https://github.com/github/statsd-
ruby)
✓node-statsd (https://github.com/sivy/node-statsd)
✓statsd-csharp-client (https://github.com/
goncalopereira/statsd-csharp-client)
✓…
Slide 110
Slide 110 text
Chapter 4
Log metrics
Slide 111
Slide 111 text
Full circle
Slide 112
Slide 112 text
Log metrics
Server 1 Server 2 Server 3
Redis
Logstash
ElasticSearch
Statsd Graphite
Slide 113
Slide 113 text
output
{
statsd
{
type
=>
"apache"
host
=>
"localhost"
port
=>
8125
namespace
=>
"logstash"
increment
=>
"apache.country.%{[geopip]
[country_code2]}"
increment
=>
"apache.response.%{response}"
count
=>
[
"apache.bytes",
"%{bytes}"
]
}
}
Logstash statsd output
Slide 114
Slide 114 text
Logstash stats
Slide 115
Slide 115 text
Recap!
Slide 116
Slide 116 text
✓Centralize your logs
✓Visualize server metrics
✓Visualize application metrics
✓Visualize log metrics
Action
Slide 117
Slide 117 text
Beaver
Redis
Logstash
ElasticSearch
CollectD
StatsD
Graphite
Graphene
Slide 118
Slide 118 text
No content