Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Speaker Deck
PRO
Sign in
Sign up for free
様々なメトリクスやログを集めてシステム解析 - Elastic Stackの入門と活用 - / Intro Elastic Stack
Jun Ohtani
July 11, 2018
0
99
様々なメトリクスやログを集めてシステム解析 - Elastic Stackの入門と活用 - / Intro Elastic Stack
OSC Hokkaido 2018での発表資料です。
Jun Ohtani
July 11, 2018
Tweet
Share
More Decks by Jun Ohtani
See All by Jun Ohtani
Elastic Stackでマイクロサービス運用を 楽にするには? / Monitoring Microservices with Elastic Stack
johtani
5
2.3k
様々なメトリクスやログを集めてシステム解析 - Elastic Stackの入門と活用 - / Getting started Elastic Stack for logging/metrics
johtani
4
780
え?SQLで入門?する ElasticsearchとElastic Stack / Getting started Elastic Stack with SQL
johtani
4
780
Elastic Stack 入門 2018.09 / Getting started Elastic Stack 2018.09
johtani
3
2.2k
What's new in Elastic Stack 6.3
johtani
2
1.7k
Elastic Stackで始めるJavaアプリのパフォーマンス監視 / Intro Elastic Stack and Elastic APM Java
johtani
5
2k
Intro Elastic Stack at Telemetry WG
johtani
0
170
What's new in Elastic Stack 6.1?
johtani
0
470
システムメトリクス・ログのリアルタイム解析入門 - Elastic Stackを活用して -
johtani
5
1.2k
Featured
See All Featured
CoffeeScript is Beautiful & I Never Want to Write Plain JavaScript Again
sstephenson
152
13k
実際に使うSQLの書き方 徹底解説 / pgcon21j-tutorial
soudai
44
14k
YesSQL, Process and Tooling at Scale
rocio
159
12k
Understanding Cognitive Biases in Performance Measurement
bluesmoon
2
390
In The Pink: A Labor of Love
frogandcode
132
21k
Optimizing for Happiness
mojombo
365
64k
GitHub's CSS Performance
jonrohan
1020
430k
Product Roadmaps are Hard
iamctodd
38
7.7k
Dealing with People You Can't Stand - Big Design 2015
cassininazir
351
21k
BBQ
matthewcrist
75
8.1k
It's Worth the Effort
3n
177
26k
Art Directing for the Web. Five minutes with CSS Template Areas
malarkey
196
9.8k
Transcript
!1 2018/07/06 Developer Advocate at Elastic Jun Ohtani @johtani ༷ʑͳϝτϦΫεϩάΛूΊͯγεςϜղੳ
- Elastic Stackͷೖͱ׆༻ -
!2
!3 ΞδΣϯμ • ϝτϦΫεʗϩάͱʁ • γεςϜϝτϦΫεղੳɺϩάղੳΛࢼ͠ʹͬͯΈΑ͏ • Beats - Elasticsearch
- KibanaͰղੳ • ຊ֨తʹղੳΛΔʹʁ • LogstashͰϩάϝτϦΫεΛதܧɾू • ͞Βʹ৭ʑࢼͯ͠ΈΔʹʁ
!4 about • Me, Jun Ohtani / Developer Advocate ‒
lucene-gosenίϛολʔ ‒ σʔλੳج൫ߏஙೖ ڞஶ ‒ http://blog.johtani.info • Elasticsearch, founded in 2012 ‒ Products: Elasticsearch, Logstash, Kibana, Beats Elastic APM, X-Pack, Elastic Cloud, Swiftype Professional services: Support & development subscriptions Trainings, Consulting, SaaS
!5 ͲΜͳϝτϦΫεɺ ϩάΛूΊ͍ͯ·͔͢ʁ
!6 ϝτϦΫε • CPUɺϝϞϦ༻ɺσΟεΫ༻ • ΞΫηεɺωοτϫʔΫసૹྔ • Ԡ࣌ؒ • ίωΫγϣϯ
• τϥϯβΫγϣϯɺച্ • ίϯςφͷ্ͷ֤छϝτϦΫε
!7 ϩά • ೝূϩά • γεςϜϩά • ΞϓϦέʔγϣϯϩά • Slow
log • ΞΫηεϩά • ίϯςφͷதͷϩά
!8 Ͱ͖ΕϩάͱϝτϦΫεΛ ·ͱΊͯ1ͭͷը໘Ͱ ݟ͍ͨͰ͢ΑͶʁ
!9 Elastic Stack
10 Elastic Stack 100% Φʔϓϯιʔε ʮΤϯλʔϓϥΠζ൛ʯແ͠ όʔδϣϯ 5.0Ͱશ౷Ұ
!11 X-Pack ؆୯ʹΠϯετʔϧ Elastic StackΛ֦ு αϒεΫϦϓγϣϯʹؚ·ΕΔ Security Alerting Monitoring Reporting
Graph Machine Learning
ఆܕͷϝτϦΫε/ϩάղੳΛ Elastic StackͰ
!13 ϝτϦΫεɾϩάͷੳʢ؆қ൛ʣ σʔλ Import Parse/ Store/Search Visualize
!14
15 Beats ܰྔσʔλγούʔ ιʔε͔ΒσʔλΛసૹ సૹ͠Elasticsearchʹू มͱύʔεͷͨΊ Logstashʹసૹ Elastic Cloudʹసૹ Libbeat:
ΧελϜbeatsͷͨ ΊͷAPIϑϨʔϜϫʔΫ 30Ҏ্ͷίϛϡχςΟbeats
The Beats family Heartbeat Uptime monitoring Filebeat Log files Winlogbeat
Windows Event Logs Packetbeat Network data +40 community Beats Metricbeat Metrics Auditbeat Audit data
Collect system and application metrics Metricbeat
lots of modules Metricbeat
tail log from file Filebeat
many modules Filebeat
Capture the Packet Packetbeat
Capture the Packet Packetbeat
Welcome to 1998 winlogbeat
Now winlogbeat
!25 • Kubernetes module in Metricbeat ‒ CPU, memory, ωοτϫʔΫసૹྔͳͲ
• add_docker_metadataϓϩηοα ‒ Container ID, name, image, labels • add_kubernetes_metadataϓϩηοα ‒ Pod name, pod namespace, container name, pod labels Beats <3 ίϯςφ DockerKubernetesͰͷσϓϩΠΛ؆୯ʹ
!26
27 Elasticsearch Heart of the Elastic Stack ࢄܕɺεέʔϥϒϧ ߴՄ༻ੑ Ϛϧνςφϯτ
։ൃऀϑϨϯυϦʔ ϦΞϧλΠϜɺશจݕࡧ ΞάϦήʔγϣϯ
Elasticsearchͱʁ
!29 ϑϦʔϫʔυݕࡧ
!30 ߜΓࠐΈ
!31 ϋΠϥΠτ
!32 ιʔτ
!33 ϖʔδϯά
!34 ूܭ
!35 αδΣετ
!36 Elasticsearch in 10 seconds • εΩʔϚϑϦʔɺࢄυΩϡϝϯτετΞɺREST & JSON •
Φʔϓϯιʔε: Apache License 2.0 • ઃఆͳ͠Ͱ؆୯ʹࢼ͢͜ͱ͕Մೳ • JavaͰ࣮ɻ֦ு༰қ
؆୯ͳCRUD
σʔλొ 38 curl -XPUT localhost:9200/books/doc/1 -d ' { "title" :
"Elasticsearch - The definitive guide", "authors" : "Clinton Gormley", "started" : "2013-02-04", "pages" : 230 }'
σʔλߋ৽ 39 curl -XPUT localhost:9200/books/doc/1 -d ' { "title" :
"Elasticsearch - The definitive guide", "authors" : [ "Clinton Gormley", "Zachary Tong" ], "started" : "2013-02-04", "pages" : 230 }'
σʔλআ !40 curl -X DELETE localhost:9200/books/doc/1 σʔλͷऔಘ curl —X GET
localhost:9200/books/doc/1 curl —X GET localhost:9200/books/doc/1/_source
ݕࡧ - Query DSL !41 curl -XGET ‘localhost:9200/books/doc/_search' -d '{
"query": { "bool": { "must": [ { "match": { "title": "Search" }}, { "match": { "content": "Elasticsearch" }} ], "filter": [ { "term": { "status": "published" }}, { "range": { "publish_date": { "gte": "2015-01-01" }}} ] } } }'
ࢄߏɺ εέʔϧ
Basic terms • ΠϯσοΫε ‒ σʔλͷཧతͳू߹ɻ RDBͷσʔλϕʔεͷΑ͏ͳͷLogical • ϨϓϦέʔγϣϯ •
ಡΈࠐΈͷεέʔϥϏϦςΟ্ • SPOFͷղফ • γϟʔσΟϯά • ෳϚγϯσʔλΛׂ ॻ͖ࠐΈͷεέʔϥϏϦςΟ্ σʔλϑϩʔ੍ޚ !43
γϟʔυͱϨϓϦΧ !44 node 1 orders products 1 4 1 2
2 3 curl -X PUT localhost:9200/orders -d '{ "settings.index.number_of_shards" : 4 "settings.index.number_of_replicas" : 1 }' curl -X PUT localhost:9200/products -d '{ "settings.index.number_of_shards" : 2 "settings.index.number_of_replicas" : 0 }'
γϟʔυͱϨϓϦΧ !45 node 1 orders products 1 4 1 node
2 orders products 2 2 3 4 1 2 3
ࣗಈతͳࢄ !46 node 1 orders products 2 1 4 1
node 2 orders products 2 2 node 3 orders products 3 4 1 3
ͦͷଞͷػೳ
elasticsearch ͞·͟·ͳܗࣜͷσʔλͰ GeoݕࡧՄೳ ҢܦɺGeoHashɺ GeoShape… GEO
Ecosystem • Plugins ‒ ϓϥάΠϯʹΑΔػೳͷՃ • ΫϥΠΞϯτϥΠϒϥϦ • Java, Ruby,
python, php, perl, javascript, .NET • Scala, clojure, go !49
Elasticsearch - The Definitive guide http://www.elastic.co/guide/en/ elasticsearch/guide/current/index.html 50 ৄ͘͠Γ͍ͨํ
!51
52 Kibana Window into the Elastic Stack ՄࢹԽͱੳ ཧۭؒ ΧελϚΠζͱ
Ϩϙʔτͷڞ༗ άϥϑ୳ࡧ Elastic Stackͷ ηΩϡΞͳΞΫηεͱཧ ΧελϜAppsͷ࡞
!53 Kibana 6
!54 σϞ σʔλೖ͔ΒՄࢹԽ·Ͱ
ຊ֨తʹղੳΛߦ͏ʹʁ
!56
57 Logstash σʔλՃύΠϓϥΠϯ શͯͷܗࣜɺαΠζͱσʔλιʔ εͷೖ ύʔεͱಈతͳ σʔλม ͋ΒΏΔग़ྗʹ σʔλసૹ ҆શͰ҉߸Խ͞Εͨ
σʔλೖྗ ಠࣗͷύΠϓϥΠϯॲཧ ͷ࡞ 200Ҏ্ͷϓϥάΠϯ
Logstash architecture !58 Input Output Filter ? ? collect and
split alter and enrich store and visualize
ઃఆ 59 input { … } filter { … }
output { … }
1ߦ1σʔλ 189.120.xx.xx - - [02/Dec/2014:12:18:29 +0900] "GET /manager/html HTTP/ 1.1"
404 274 "-" "Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0" 60
ઃఆɿfilter 61 filter { grok { match => { "message"
=> "%{COMBINEDAPACHELOG}" } break_on_match => false } date { match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"] locale => en } geoip { source => ["clientip"] } useragent { source => "agent" target => "useragent" } }
ύʔε !62 189.120.xx.xx - - [02/Dec/2014:12:18:29 +0900] "GET /manager/html HTTP/1.1"
404 274 "-" "Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0" {… "@timestamp": "2015-04-10T09:07:49.325Z", "clientip": "189.120.xx.xx", "ident": "-", "auth": "-", "timestamp": "02/Dec/2014:12:18:29 +0900", "verb": "GET", "request": "/manager/html", … "agent": "\"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/
ઃఆɿfilter !63 filter { grok { match => { "message"
=> "%{COMBINEDAPACHELOG}" } break_on_match => false } date { match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"] locale => en } geoip { source => ["clientip"] } useragent { source => "agent" target => "useragent" } }
ͷύʔε 64 {… "@timestamp": "2015-04-10T09:07:49.325Z", … "timestamp": "02/Dec/2014:12:18:29 +0900", …
} {… "@timestamp": "2014-12-02T03:18:29.000Z", … "timestamp": "02/Dec/2014:12:18:29 +0900", … }
ઃఆɿfilter !65 filter { grok { match => { "message"
=> "%{COMBINEDAPACHELOG}" } break_on_match => false } date { match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"] locale => en } geoip { source => ["clientip"] } useragent { source => "agent" target => "useragent" } }
IP͔ΒҢܦͳͲ༩ 66 "clientip": "189.120.xx.xx", "clientip": "189.120.xx.xx", "geoip": { "ip": “189.120.xxx.xxx”,
… "country_name": "Brazil", "continent_code": "SA", "region_name": "27", "city_name": "São Paulo", "latitude":
ઃఆɿfilter !67 filter { grok { match => { "message"
=> "%{COMBINEDAPACHELOG}" } break_on_match => false } date { match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"] locale => en } geoip { source => ["clientip"] } useragent { source => "agent" target => "useragent" } }
ϢʔβΤʔδΣϯτͷύʔε 68 "agent": "\"Mozilla/5.0 (Windows NT 5.1; rv: 5.0) Gecko/20100101
Firefox/5.0\"" "agent": "\"Mozilla/5.0 (Windows NT 5.1; rv: 5.0) Gecko/20100101 Firefox/5.0\"" "useragent": { "name": "Firefox", "os": "Windows XP", "os_name": "Windows XP", "device": "Other", "major": "5", "minor": "0"
ͦͷ΄͔ʹʁ
!70 elasticsearch-hadoop - • D E H • PD ecd
ER • g D • CH • Ca M DMS D FERC
!71
!72 X-Pack ؆୯ʹΠϯετʔϧ Elastic StackΛ֦ு αϒεΫϦϓγϣϯʹؚ·ΕΔ Security Alerting Monitoring Reporting
Graph Machine Learning
!73
!74 ࢀߟจݙ • Elasticsearch - The Definitive guide ‒ http://www.elastic.co/guide/en/elasticsearch/guide/current/
index.html • ॻ੶ʢຊޠʣ ‒ σʔλੳج൫ߏஙೖ ‒ Elasticsearch࣮ફΨΠυ
!75 ࢀߟαΠτ • Ϣʔεέʔε • https://www.elastic.co/use-cases • DiscussʢWebϑΥʔϥϜʣ • https://discuss.elastic.co
• Elastic{ON}ͷϏσΦͱࢿྉ • https://www.elastic.co/elasticon/videos • αϙʔτϝχϡʔ • https://www.elastic.co/subscriptions
Thanks for listening! Q & A We’re hiring! https://www.elastic.co/about/careers/ We’re
helping! https://www.elastic.co/subscriptions http://training.elastic.co