Data Discovery and Systems Diagnostics with the ELK stack

[email protected] www.rittmanmead.com @rittmanmead
Robin Moffatt, Principal Consultant, Rittman Mead | YoDB November 2015
Data Discovery & Systems Diagnostics
with the ELK Stack
1

View Slide

• Principal Consultant with Rittman Mead

- OBIEE & ODI / SysAdmin / Performance
• Previously …

- OBIEE/DW developer at large UK retailer
- SQL Server DBA, BusinessObjects, DB2, COBOL…. 
• Blog: http://ritt.md/rmoff

• Twitter: @rmoff

• IRC: rmoff / #obihackers / freenode
Robin Moffatt
2

View Slide

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or  
+61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected]
W : www.rittmanmead.com
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or  
ELK

View Slide

ELK
4
• Elasticsearch - schema-free, distributed data store

• Logstash - centralised data processing

• Kibana - analytics and visualisation

View Slide

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or  
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or  
Data
Discovery

View Slide

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or  
E : [email protected]ead.com
6
ELK - Data Discovery

View Slide

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or  
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or  
Systems
Diagnostics

View Slide

ELK - Systems Diagnostics
8

View Slide

Getting started is easy!
9
1. Download
2. Unarchive
3. Run
4. …

5. erm

6. that’s it!

View Slide

[email protected] www.rittmanmead.com @rittmanmead 10
logstash
Elasticsearch
Kibana
logs

View Slide

DEMO!
11

View Slide

Logstash
Elasticsearch
Kibana
twitter
csv

View Slide

DEMO!
13

View Slide

Elasticsearch
14
• The core component of the ELK stack
• Based on Apache Lucene (same as Cloudera’s Solr)

• Distributed for scalability & resilience

• Near-realtime document indexing

View Slide

Elasticsearch Uses - Search and Analytics
15
• Search
- Soundcloud
- GitHub 
• Analytics
- The Guardian’s Ophan application
https://www.elastic.co/assets/bltd061cc55096a5780/case-study-the-guardian.pdf
A quarter of a billion
events per day …
typically the lag before
something shows up on
the dashboard is
somewhere between
three to ﬁve seconds…
http://tnw.to/s3NV5

View Slide

Elasticsearch
16
• Stores data as JSON documents
within an index

• An index is made up of shards

• Shards are distributed around a
cluster automatically
- Resilience and scale-out are simple

View Slide

Elasticsearch Administration
17
https://github.com/lmenezes/elasticsearch-kopf
Kopf
Marvel

View Slide

Working with the REST API
18
• curl is all you need ;-)

- optionally with jq for syntax highlighting of the resulting json
• Sense is a free plugin from Elastic for Kibana and useful for
rapid prototyping of more complex interactions

View Slide

Elasticsearch REST API
19
$ curl -XPOST  
'http://es:9200/viz/characters/'
-d '{"name":"finbarr saunders"}'

View Slide

$ curl -XPOST
'http://es:9200/viz/characters/'
-d '{"name":"roger mellie”,
"notes":"the man on the tele"}'
20

View Slide

$ curl -XGET 'http://localhost:9200/viz/_search?q=roger'
[…]
"hits" : { "total" : 1, "max_score" : 0.11506981,
"hits" : [ {
"_index" : "viz",
"_type" : "characters",
"_id" : "AUyyNUrTI0Rm5Pb-t8_l",
"_score" : 0.11506981,
"_source":{"name":"roger mellie" ,"notes":"the man on  
the tele"}
21

View Slide

$ curl -XDELETE  
'http://localhost:9200/viz'
22

View Slide

Elasticsearch-Hadoop
23
• Two-way connector between Hadoop
and Elasticsearch

• Read/Write with Elasticsearch from
Hive, Pig, Spark, etc
https://www.elastic.co/products/hadoop
Hive
HDFS
Elasticsearch
Tweets Website logs Blog post metadata
Flume CSV
elasticsearch-hadoop
Kibana
http://ritt.md/elk-hadoop-01
Logstash

View Slide

Logstash
24
input
filter
output
elasticsearch email
kafka
nagios
pagerduty stdout file
grok geoip mutate drop
kafka
log csv
tsv
json
syslog tcp jdbc
stdin
twitter

View Slide

Logstash
25
• Does Logstash support …. yes, probably!

- Vast number of supported input (and output) formats
couchdb_changes
drupal_dblog
elasticsearch
exec
eventlog
file
ganglia
gelf
generator
graphite
github
heartbeat
heroku
irc
imap
jmx
kafka
log4j
lumberjack
meetup
pipe
puppet_facter
relp
rss
rackspace
rabbitmq
redis
snmptrap
stdin
sqlite
s3
sqs
stomp
syslog
tcp
twitter
unix
udp
varnishlog
wmi
websocket
xmpp
zenoss
zeromq
Outputs
boundary
circonus
csv
cloudwatch
datadog
datadog_metrics
email
elasticsearch
exec
file
google_bigquery
google_cloud_storage
ganglia
gelf
graphtastic
graphite
hipchat
http
irc
influxdb
juggernaut
jira
kafka
lumberjack
librato
loggly
mongodb
metriccatcher
nagios
null
nagios_nsca
opentsdb
pagerduty
pipe
riemann
redmine
rackspace
rabbitmq
redis
riak
s3
sqs
stomp
statsd
solr_http
sns
syslog
stdout
tcp
udp
websocket
xmpp
zabbix
zeromq
Inputs
http://www.elastic.co/guide/en/logstash/master/input-plugins.html http://www.elastic.co/guide/en/logstash/master/output-plugins.html

View Slide

Logstash Filters
26
• Powerful data processing

- Extract fields from input
(grok)
- Enrich data (geoip, dns)
- Reformat (csv, split, multiline,
json, xml)
alter
anonymize
collate
csv
cidr
clone
cipher
checksum
date
dns
drop
elasticsearch
extractnumbers
environment
elapsed
fingerprint
geoip
grok
i18n
json
json_encode
kv
mutate
metrics
multiline
metaevent
prune
punct
ruby
range
syslog_pri
sleep
split
throttle
translate
uuid
urldecode
useragent
xml
zeromq
Filters
http://www.elastic.co/guide/en/logstash/master/ﬁlter-plugins.html

View Slide

Grok — Time to get your RegEx on!
27
Input
data
Grok
pattern
Key/Value output
http://grokdebug.herokuapp.com/

View Slide

Logstash in Action
28
filter { grok { match => [ "message",
"\[%{TIMESTAMP_ISO8601:timestamp}\]  
\[%{DATA:Component}\]  
\[%{WORD:Severity}
(:%{NUMBER:LogLevelNum})?\]
input {file { path => ["nqserver.log" ] }}
output {
elasticsearch { host => "localhost" }}

View Slide

Logstash -> Elasticsearch
29

View Slide

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or  
About Rittman Mead
30
• Oracle BI and DW Gold partner

•Winner of five UKOUG Partner of the Year awards in 2013 and 2014 - including BI
• World leading specialist partner for technical excellence,  
solutions delivery and innovation in Oracle BI

• Approximately 80 consultants worldwide

• All expert in Oracle BI and DW

• Offices in US (Atlanta), Europe, Australia and India

• Skills in broad range of supporting Oracle tools:

- OBIEE, OBIA
- ODIEE
- Essbase, Oracle OLAP
- GoldenGate
- Endeca
Systems
Diagnostics 
with ELK

View Slide


View Slide

System Diagnostics
32

View Slide

System Monitoring
33

View Slide

System Monitoring
34

View Slide

Performance Diagnostics
35

View Slide

#EOF
36
email 
[email protected]
web 
http://ritt.md/rmoff
twitter 
@rmoff
irc 
rmoff @ #obihackers
Data Discovery
http://ritt.md/go-elk-1
System Diagnostics & Monitoring
Logstash & Kafka
http://ritt.md/kafka-elk 
http://ritt.md/kafka-pipelines

View Slide

Data Discovery and Systems Diagnostics with the ELK stack

Data Discovery and Systems Diagnostics with the ELK stack

Robin Moffatt

More Decks by Robin Moffatt

Other Decks in Technology

Featured

Transcript