Slide 1
Slide 1 text
‹#›
2017/09/25
Evangelist at Elastic
Jun Ohtani @johtani
Elastic StackʹΑΔ
ϩάͷࢹೖ
Slide 2
Slide 2 text
‹#›
Slide 3
Slide 3 text
ΞδΣϯμ
• Elastic Stack ͱʁ
• ؆୯ͳհ
• Elastic StackͰ͍Ζ͍ΖϩάՄࢹԽʴࢹ
• Audit log / Access log / MySQL slow log
• ΧελϜϩά
• ͞Βʹͦͷઌʁ
3
Slide 4
Slide 4 text
about
• Me, Jun Ohtani / Technical Advocate
‒ lucene-gosenίϛολʔ
‒ ElasticSearch Serverຊޠ൛ͷ༁
‒ http://blog.johtani.info
• Elasticsearch, founded in 2012
‒ Products: Elasticsearch, Logstash, Kibana, Beats
X-Pack, Elastic Cloud
Professional services: Support & development subscriptions
‒ Trainings, Consulting, SaaS
4
Slide 5
Slide 5 text
5
Elastic Stack
100% open source
No enterprise edition
All new versions with 5.0
Slide 6
Slide 6 text
6
Beats
Lightweight Data Shippers
Ship data from the source
Ship and centralize in
Elasticsearch
Ship to Logstash for
transformation and parsing
Ship to Elastic Cloud Libbeat: API framework to
build custom beats 30+ community Beats
Slide 7
Slide 7 text
7
FILEBEAT
Log Files
METRICBEAT
Metrics
PACKETBEAT
Network Data
WINGLOGBEAT
Window Events
More than 30 community Beats
and growing …
Apachebeat, dockbeat, httpbeat,
mysqlbeat, nginxbeat, redis beats,
twitterbeat, and more
Slide 8
Slide 8 text
8
Logstash
Data processing pipeline
Ingest data of all shapes,
sizes, and sources
Parse and dynamically
transform data
Transport data to any
output
Secure and encrypt data
inputs
Build your own pipeline More than 200+ plugins
Slide 9
Slide 9 text
Logstash architecture
9
Input Output
Filter
? ?
collect and split alter and enrich store and visualize
Slide 10
Slide 10 text
Elasticsearch
Heart of the Elastic Stack
10
Distributed, Scalable High-availability Multi-tenancy
Developer Friendly Real-time, Full-text Search Aggregations
Slide 11
Slide 11 text
ݕࡧͱͯ͠ͷ
Elasticsearch
Slide 12
Slide 12 text
Elasticsearchͱʁ
Slide 13
Slide 13 text
ϑϦʔϫʔυݕࡧ
13
Slide 14
Slide 14 text
ߜΓࠐΈ
14
Slide 15
Slide 15 text
ϋΠϥΠτ
15
Slide 16
Slide 16 text
ιʔτ
16
Slide 17
Slide 17 text
ϖʔδϯά
17
Slide 18
Slide 18 text
ूܭ
18
Slide 19
Slide 19 text
αδΣετ
19
Slide 20
Slide 20 text
Elasticsearch in 10 seconds
• εΩʔϚϑϦʔɺࢄυΩϡϝϯτετΞɺREST & JSON
• Φʔϓϯιʔε: Apache License 2.0
• ઃఆͳ͠Ͱ؆୯ʹࢼ͢͜ͱ͕Մೳ
• JavaͰ࣮ɻ֦ு༰қ
20
Slide 21
Slide 21 text
σʔλొ
21
curl -XPUT localhost:9200/books/book/1 -d '
{
"title" : "Elasticsearch - The definitive guide",
"authors" : "Clinton Gormley",
"started" : "2013-02-04",
"pages" : 230
}'
Slide 22
Slide 22 text
Ecosystem
• Plugins
‒ ϓϥάΠϯʹΑΔػೳͷՃ
• ΫϥΠΞϯτϥΠϒϥϦ
• Java, Ruby, python, php, perl, javascript, .NET
• Scala, clojure, go
22
Slide 23
Slide 23 text
23
Kibana
Window into the Elastic Stack
Visualize and analyze Geospatial Customize and Share
Reports
Graph Exploration UX to secure and manage
the Elastic Stack
Build Custom Apps
Slide 24
Slide 24 text
24
Slide 25
Slide 25 text
25
Slide 26
Slide 26 text
No content
Slide 27
Slide 27 text
Nginx/Apache2 Access Log
Slide 28
Slide 28 text
A Nginx/Apache2 module - Filebeat
• Enable Nginx/Apache2 module
• Configure only path of access log file
• Also error file
• Run filebeat
• Load Kibana dashboard
• Tutrial
https://www.elastic.co/guide/en/beats/
filebeat/current/_tutorial.html
28
Slide 29
Slide 29 text
Auditd log
Slide 30
Slide 30 text
Auditd module - Filebeat
• Enable auditd module
• Configure only path of audit log file
• Run filebeat
• Load Kibana dashboard
30
Slide 31
Slide 31 text
Deep dive in Filebeat module
1. Filebeat : Load index template to Elasticsearch if it doesn't exist
2. Filebeat : Load ingest pipeline setting to Es if it doesn't exist
3. Filebeat : Load Kibana sample dashboard
4. Filebeat : Ship log data to Elasticsearch with pipeline parameter
5. Elasticsearch : Parse/Modify/Enrich logs with ingest node in Es
6. Elasticsearch : Store logs in Es
31
Slide 32
Slide 32 text
MySQL slow log
Slide 33
Slide 33 text
MySQL module - Filebeat
• Enable mysql module
• Configure only path of slow log file
• Also error log file
• Run filebeat
• Load Kibana dashboard
33
Slide 34
Slide 34 text
Packetbeat - Another monitoring
• Network packet analyzer
• Support many protocols
• Ship to Es or Logstash
• Visualize in Kibana
• Products page
https://www.elastic.co/
products/beats/packetbeat
34
Slide 35
Slide 35 text
Sample dashboard and demo
35
Slide 36
Slide 36 text
ΧελϜͳLog
Slide 37
Slide 37 text
Multi line
• Support Filebeat & Logstash
37
Filebeat Logstash
Slide 38
Slide 38 text
Grok filter
38
filter {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
break_on_match => false
}
date {
match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"]
locale => en
}
geoip { source => ["clientip"] }
useragent {
source => "agent"
target => "useragent"
}
}
Slide 39
Slide 39 text
Parse with Grok filter
39
189.120.xx.xx - - [02/Dec/2014:12:18:29 +0900] "GET /manager/html HTTP/1.1"
404 274 "-" "Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0"
{…
"@timestamp": "2015-04-10T09:07:49.325Z",
"clientip": "189.120.xx.xx",
"ident": "-",
"auth": "-",
"timestamp": "02/Dec/2014:12:18:29 +0900",
"verb": "GET",
"request": "/manager/html",
…
"agent": "\"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/
Slide 40
Slide 40 text
Grok patterns
• Regular expression & Allow you to name it
• Over 120 reusable patterns
• Example pattern:
40
USERNAME [a-zA-Z0-9._-]+
INT (?:[+-]?(?:[0-9]+))
COMMONAPACHELOG %{IPORHOST:clientip} %
{HTTPDUSER:ident} %{USER:auth}
\[%{HTTPDATE:timestamp}\] "(?:%
{WORD:verb} %
{NOTSPACE:request}(?: HTTP/%
Slide 41
Slide 41 text
Grok debugger
• In Kibana dev tools since 5.5.0 with X-Pack basic
• Still provide https://grokdebug.herokuapp.com
41
Slide 42
Slide 42 text
͞Βʹͦͷઌʁ
Slide 43
Slide 43 text
Single install
Extensions for the Elastic Stack
Subscription pricing
X-Pack
43
Security
Alerting
Monitoring
Reporting
Graph
Machine Learning
Slide 44
Slide 44 text
Ξϥʔτͱ༧ଌ
Slide 45
Slide 45 text
Alerting
X-Pack
45
• Create Watches to detect changes in
your data
• Trigger automatic notifications
• Setup nested alerts
• Store and track alert history
SETUP ALERTS
NOTIFY AND INTEGRATE
• Email
• Slack
• Pagerduty
• Hipchat or JIRA
• Other monitoring systems
Slide 46
Slide 46 text
21
Slide 47
Slide 47 text
Machine Learning
X-Pack
47
• Automatically detect anomalies
• Advanced correlation and categorization
• Identify root cause(s)
• Expose early warning signs
UNSUPERVISED MACHINE LEARNING
ENABLE NEW USE CASES
• Analyze time series data
• Expand security, IT Ops, fraud, finance, and
many more use cases
• Available as beta in the 5.4 release
Slide 48
Slide 48 text
48
Slide 49
Slide 49 text
ࢀߟจݙ
• Elasticsearch - The Definitive guide
‒ http://www.elastic.co/guide/en/elasticsearch/guide/current/index.html
• ॻ੶ʢຊޠʣ
‒ ElasticSearchServerຊޠ൛
‒ σʔλੳج൫ߏஙೖ
20179݄21ൃച
49
Slide 50
Slide 50 text
ࢀߟαΠτ
• Ϣʔεέʔε
• https://www.elastic.co/use-cases
• DiscussʢWebϑΥʔϥϜʣ
• https://discuss.elastic.co
• Elastic{ON}ͷϏσΦͱࢿྉ
• https://www.elastic.co/elasticon/videos
• αϙʔτϝχϡʔ
• https://www.elastic.co/subscriptions
50
Slide 51
Slide 51 text
Thanks for listening!
Q & A
We’re hiring!
https://www.elastic.co/about/careers/
We’re helping!
https://www.elastic.co/subscriptions
http://training.elastic.co