様々なメトリクスやログを集めてシステム解析 - Elastic Stackの入門と活用 - / Getting started Elastic Stack for logging/metrics

1.

!1 2018/10/27 Community Engineer @Elastic  Jun Ohtani @johtani ༷ʑͳϝτϦΫε΍ϩάΛूΊͯγεςϜղੳ  
- Elastic Stackͷೖ໳ͱ׆༻ -

2.

!2 ΞδΣϯμ • ϝτϦοΫʗϩάͱ͸ʁ • γεςϜϝτϦΫεղੳɺϩάղੳΛࢼ͠ʹ΍ͬͯΈΑ͏ • Beats - Elasticsearch
- KibanaͰղੳ • ຊ֨తʹղੳΛ΍Δʹ͸ʁ • LogstashͰϩά΍ϝτϦΫεΛதܧɾू໿ • ͞Βʹ৭ʑࢼͯ͠ΈΔʹ͸ʁ

3.

!3 about • Me, Jun Ohtani / Community Engineer ‒
lucene-gosenίϛολʔ ‒ σʔλ෼ੳج൫ߏஙೖ໳ ڞஶ ‒ http://blog.johtani.info  • Elastic, founded in 2012 ‒ Products: Elasticsearch, Logstash, Kibana, Beats   Elastic APM,   Elastic Cloud, Swiftype   Professional services: Support & development subscriptions  Trainings, Consulting, SaaS

4.

!4 ͲΜͳϝτϦοΫɺ  ϩάΛूΊ͍ͯ·͔͢ʁ

5.

!5 ϝτϦοΫ • CPUɺϝϞϦ࢖༻཰ɺσΟεΫ࢖༻཰ • ΞΫηε਺ɺωοτϫʔΫసૹྔ • Ԡ౴࣌ؒ • ίωΫγϣϯ਺
• τϥϯβΫγϣϯ਺ɺച্ • ίϯςφͷ্ͷ֤छϝτϦΫε

6.

!6 ϩά • ೝূϩά • γεςϜϩά • ΞϓϦέʔγϣϯϩά • Slow
log • ΞΫηεϩά • ίϯςφͷதͷϩά

7.

!7 Ͱ͖Ε͹ϩάͱϝτϦοΫΛ  ·ͱΊͯ1ͭͷը໘Ͱ  ݟ͍ͨͰ͢ΑͶʁ

8.

!8 Elastic Stack

9.

Elastic Stack อଘɺݕࡧɺ෼ੳ Elasticsearch ՄࢹԽɺ؅ཧ Kibana Beats ΠϯδΣετ Logstash

10.

Metrics Logging APM Site  Search Application Search Business  Analytics Enterprise 
Search Security  Analytics Future ιϦϡʔγϣϯ อଘɺݕࡧɺ෼ੳ ՄࢹԽɺ؅ཧ ΠϯδΣετ Kibana Elasticsearch Beats Logstash Elastic Stack

11.

Metrics Logging APM Site  Search App  Search Business  Analytics Enterprise 
Search Security  Analytics Future ιϦϡʔγϣϯ SaaS Elastic Cloud Self Managed Elastic Cloud  Enterprise Standalone σϓϩΠ อଘɺݕࡧɺ෼ੳ ՄࢹԽɺ؅ཧ ΠϯδΣετ Kibana Elasticsearch Beats Logstash Elastic Stack

12.

อଘɺݕࡧɺ෼ੳ Elasticsearch ՄࢹԽɺ؅ཧ Kibana Beats ΠϯδΣετ Logstash Metrics Logging APM
Site  Search Application Search Business  Analytics Enterprise  Search Security  Analytics Future ιϦϡʔγϣϯ SaaS Elastic Cloud Self Managed Elastic Cloud  Enterprise Standalone σϓϩΠ Elastic Stack

13.

!13 ఆܕͷϝτϦΫε/ϩάղੳΛ Elastic StackͰ

14.

!14 ϝτϦοΫɾϩά෼ੳʢ؆қ൛ʣ Beats Log Files Metrics Wire Data Kibana Instances
Elasticsearch Nodes

15.

!15

16.

16 Beats ܰྔσʔλγούʔ ιʔε͔ΒσʔλΛసૹ సૹ͠Elasticsearchʹू໿ ม׵ͱύʔεͷͨΊ Logstashʹసૹ Elastic Cloudʹసૹ Libbeat:
ΧελϜbeatsͷͨ ΊͷAPIϑϨʔϜϫʔΫ 30Ҏ্ͷίϛϡχςΟbeats

17.

The Beats family Heartbeat Uptime monitoring Filebeat Log files Winlogbeat
Windows Event Logs Packetbeat Network data +40 community Beats Metricbeat Metrics Auditbeat Audit data

18.

Collect system and application metrics Metricbeat

19.

lots of modules Metricbeat

20.

!20 Metricbeat Ϟδϡʔϧ • Aerospike module • Apache module •
Ceph module • Couchbase module • Docker module • Dropwizard module • Elasticsearch module • Etcd module • Golang module • Graphite module • HAProxy module • HTTP module • Jolokia module • Kafka module • Kibana module • Kubernetes module • kvm module • Logstash module • Memcached module • MongoDB module • Munin module • MySQL module • Nginx module • • PHP_FPM module • PostgreSQL module • Prometheus module • RabbitMQ module • Redis module • System module • uwsgi module • vSphere module • Windows module • ZooKeeper module 

21.

tail log from ﬁle Filebeat

22.

many modules Filebeat

23.

Filebeat modules - v6.4.2 • Apache2 module • Auditd module
• Icinga module • IIS module • Kafka module • Logstash module • MongoDB module • MySQL module • Nginx module • Osquery module • PostgreSQL module • Redis module • System module • Traefik module

24.

Capture the Packet Packetbeat

25.

Capture the Packet Packetbeat

26.

Welcome to 1998 winlogbeat

27.

Now winlogbeat

28.

!28

29.

29 Elasticsearch Heart of the Elastic Stack ෼ࢄܕɺεέʔϥϒϧ ߴՄ༻ੑ Ϛϧνςφϯτ
։ൃऀϑϨϯυϦʔ ϦΞϧλΠϜɺશจݕࡧ ΞάϦήʔγϣϯ

30.

Elasticsearchͱ͸ʁ

31.

ϑϦʔϫʔυݕࡧ !31

32.

ߜΓࠐΈ !32

33.

ϋΠϥΠτ !33

34.

ιʔτ !34

35.

ϖʔδϯά !35

36.

ूܭ !36

37.

αδΣετ !37

38.

Elasticsearch in 10 seconds • εΩʔϚϑϦʔɺ෼ࢄυΩϡϝϯτετΞɺREST & JSON • Φʔϓϯιʔε:
Apache License 2.0 • ઃఆͳ͠Ͱ؆୯ʹࢼ͢͜ͱ͕Մೳ • JavaͰ࣮૷ɻ֦ு΋༰қ !38

39.

؆୯ͳCRUD

40.

σʔλొ࿥ 40 curl -XPUT localhost:9200/books/book/1 -d ' { "title" :
"Elasticsearch - The definitive guide", "authors" : "Clinton Gormley", "started" : "2013-02-04", "pages" : 230 }'

41.

σʔλߋ৽ 41 curl -XPUT localhost:9200/books/book/1 -d ' { "title" :
"Elasticsearch - The definitive guide", "authors" : [ "Clinton Gormley", "Zachary Tong" ], "started" : "2013-02-04", "pages" : 230 }'

42.

σʔλ࡟আ !42 curl -X DELETE localhost:9200/books/book/1 σʔλͷऔಘ curl —X GET
localhost:9200/books/book/1 curl —X GET localhost:9200/books/book/1/_source

43.

ݕࡧ - Query DSL !43 curl -XGET ‘localhost:9200/books/doc/_search' -d '{
"query": { "bool": { "must": [ { "match": { "title": "Search" }}, { "match": { "content": "Elasticsearch" }} ], "filter": [ { "term": { "status": "published" }}, { "range": { "publish_date": { "gte": "2015-01-01" }}} ] } } }'

44.

෼ࢄߏ੒ɺ  εέʔϧ

45.

Basic terms • ΠϯσοΫε ‒ σʔλͷ࿦ཧతͳू߹ɻ  RDBͷσʔλϕʔεͷΑ͏ͳ΋ͷLogical • ϨϓϦέʔγϣϯ •
ಡΈࠐΈͷεέʔϥϏϦςΟ޲্ • SPOFͷղফ • γϟʔσΟϯά • ෳ਺Ϛγϯ΁σʔλΛ෼ׂ  ॻ͖ࠐΈͷεέʔϥϏϦςΟ޲্  σʔλϑϩʔ੍ޚ !45

46.

γϟʔυͱϨϓϦΧ !46 node 1 orders products 1 4 1 2
2 3 curl -X PUT localhost:9200/orders -d '{ "settings.index.number_of_shards" : 4 "settings.index.number_of_replicas" : 1 }' curl -X PUT localhost:9200/products -d '{ "settings.index.number_of_shards" : 2 "settings.index.number_of_replicas" : 0 }'

47.

γϟʔυͱϨϓϦΧ !47 node 1 orders products 1 4 1 node
2 orders products 2 2 3 4 1 2 3

48.

ࣗಈతͳ෼ࢄ !48 node 1 orders products 2 1 4 1
node 2 orders products 2 2 node 3 orders products 3 4 1 3

49.

શจݕࡧͱ͸ʁ

50.

શจݕࡧͱ͸ʁ • શจݕࡧʢFull text searchʣͱ͸ɺίϯϐϡʔλʹ͓͍ͯɺෳ਺ͷจॻ ʢϑΝΠϧʣ͔ΒಛఆͷจࣈྻΛݕࡧ͢Δ͜ͱɻʮϑΝΠϧ໊ݕࡧʯ΍ ʮ୯ҰϑΝΠϧ಺ͷจࣈྻݕࡧʯͱҟͳΓɺʮෳ਺จॻʹ·͕ͨͬͯɺจ ॻʹؚ·ΕΔશจΛର৅ͱͨ͠ݕࡧʯͱ͍͏ҙຯͰ࢖༻͞ΕΔɻ  ʢWikipediaΑΓʣ !50

51.

༻ޠ • ΠϯσοΫε ݕࡧΤϯδϯ͕ݕࡧʹ࢖༻͢Δσʔλͷอଘઌ • υΩϡϝϯτʢจॻʣ ‒ ݕࡧΤϯδϯʹอଘ͞Εͨσʔλ • ϑΟʔϧυ
‒ υΩϡϝϯτʹؚ·ΕΔଐੑ • ΫΤϦ ‒ ݕࡧ৚݅ɺݕࡧࣜ !51

52.

༻ޠ • εΩʔϚ ‒ υΩϡϝϯτͷߏ଄Λఆٛ͢Δ΋ͷ • λʔϜʢTermʣɺτʔΫϯʢTokenʣ ‒ ΠϯσοΫεͷΩʔʹͳΔ୯ޠʢจࣈྻʣ ‒
จষΛҰఆͷ๏ଇͰ۠੾ͬͨ୯ޠ ‒ ୯ޠ͚ͩͰͳ͘ɺ୯ޠͷҐஔͳͲ΋ؚΉ !52

53.

υΩϡϝϯτͷొ࿥ !53 1 2 ΧπΦ͸αβΤͷఋ αβΤ͸ϫΧϝͷ࢞ υΩϡϝϯτͷొ࿥

54.

υΩϡϝϯτͷొ࿥ !54 1 2 ΧπΦ͸αβΤͷఋ αβΤ͸ϫΧϝͷ࢞ 1 2 ΧπΦ αβΤ
͸ ͸ ͷ ͷ αβΤ ϫΧϝ ఋ ࢞ υΩϡϝϯτͷొ࿥ ୯ޠʹ෼ׂ

55.

υΩϡϝϯτͷొ࿥ !55 1 2 ΧπΦ͸αβΤͷఋ αβΤ͸ϫΧϝͷ࢞ 1 2 ΧπΦ αβΤ
͸ ͸ ͷ ͷ αβΤ ϫΧϝ ఋ ࢞ ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 υΩϡϝϯτͷొ࿥ ୯ޠʹ෼ׂ ୯ޠ͔Βidͷ഑ྻ͕ Ҿ͚ΔΑ͏ʹ

56.

ݕࡧ !56 ΧπΦ αβΤ 2 ͸ ͷ ࢞ ϫΧϝ 2
1 2 1 2 1 ఋ 2 ݕࡧ৚݅ೖྗ ΧπΦɹαβΤ 1 1

57.

1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ৚݅ೖྗ ݕࡧ৚݅ͷύʔε  ݕࡧΫΤϦԽ ΧπΦɹαβΤ 1 1

58.


59.


60.

ݕࡧ !60 ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞
ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ৚݅ೖྗ ݕࡧ৚݅ͷύʔε  ݕࡧΫΤϦԽ ΧπΦɹαβΤ

61.

ݕࡧ !61 ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞

62.

ݕࡧ !62 ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞

63.

୯ޠͷ۠੾Γํ • ӳޠͷ৔߹ I am speaking Introduction Elasticsearch.    
• ೔ຊޠͷ৔߹ ࢲ͸ೖ໳Elasticsearchʹ͍ͭͯ࿩͍ͯ͠Δɻ    !63

64.

୯ޠͷ۠੾Γํ • ӳޠͷ৔߹ I am speaking Introduction Elasticsearch.    
εϖʔε͕੾Ε໨ͱΘ͔Δ • ೔ຊޠͷ৔߹ ࢲ͸ೖ໳Elasticsearchʹ͍ͭͯ࿩͍ͯ͠Δɻ  Ͳ͜Ͱ۠੾Ε͹Α͍ʁ 64

65.

N-Gramͱܗଶૉղੳ • సஔΠϯσοΫεͷΩʔͷ࡞Γํ ‒ ೔ຊޠ͸୯ޠͷ੾Ε໨͕Θ͔Βͳ͍ͷͰɺసஔΠϯσοΫεͷΩʔ͸ ओʹ࣍ͷ̎ͭͷख๏Ͱ࡞੒ • N-Gram ‒ NจࣈͣͭจষΛ۠੾Δ
• ܗଶૉղੳ ‒ ࣙॻͳͲΛ༻͍ͯҙຯͷ͋Δ୯ޠͰ۠੾Δ !65

66.

ܗଶૉղੳ • ϝϦοτɿ ‒ ҙຯͷ͋Δ୯ޠͷ੾Ε໨  ඼ࢺ৘ใΛݩʹ௥Ճॲཧ͕Մೳʢޠװม׵ͳͲʣ • σϝϦοτɿ ‒ ৽ޠʢະ஌ޠʣʹऑ͍→ࣙॻϕʔεͷ৔߹ɺࣙॻʹͳ͍୯ޠ͸ݕग़ෆ
ೳɻ !66 ΧπΦ͸αβΤͷఋ ΧπΦ ͸ ͷ αβΤ ఋ

67.

N-Gram • ϝϦοτɿ ‒ ະ஌ޠʹରԠՄೳ • σϝϦοτɿ ‒ ΠϯσοΫεංେԽ ‒
඼ࢺ৘ใʹجͮ͘ॲཧ͕ෆՄೳ !67 ΧπΦ͸αβΤͷఋ Χπ πΦ Φ͸ ͸α αβ βΤ Τͷ ͷఋ

68.

ͦͷଞͷػೳ

69.

elasticsearch ͞·͟·ͳܗࣜͷσʔλͰ GeoݕࡧՄೳ    Ң౓ܦ౓ɺGeoHashɺ GeoShape… GEO

70.

Ecosystem • Plugins ‒ ϓϥάΠϯʹΑΔػೳͷ௥Ճ • ΫϥΠΞϯτϥΠϒϥϦ • Java, Ruby,
python, php, perl, javascript, .NET • Scala, clojure, go !70

71.

Elasticsearch - The Definitive guide    http://www.elastic.co/guide/en/ elasticsearch/guide/current/index.html 71 ৄ͘͠஌Γ͍ͨํ͸

72.

!72

73.

73 Kibana Window into the Elastic Stack ՄࢹԽͱ෼ੳ ஍ཧۭؒ ΧελϚΠζͱ
Ϩϙʔτͷڞ༗ άϥϑ୳ࡧ Elastic Stack΁ͷ ηΩϡΞͳΞΫηεͱ؅ཧ ΧελϜAppsͷ࡞੒

74.

!74 Kibana 6

75.

!75 σϞ σʔλ౤ೖ͔ΒՄࢹԽ·Ͱ

76.

!76 ຊ֨తʹղੳΛߦ͏ʹ͸ʁ

77.

!77 Elastic Stackͷߏ੒ Beats Log Files Metrics Wire Data Kibana
Instances Elasticsearch Nodes

78.

!78 Elastic Stackͷߏ੒ Beats Log Files Metrics Wire Data your{beat}
Kibana Instances Kafka Distributed Message Queue Notification Queues Storage Metrics Data Store Web APIs Social Sensors Elasticsearch Nodes Logstash Nodes

79.

!79

80.

80 Logstash σʔλՃ޻ύΠϓϥΠϯ શͯͷܗࣜɺαΠζͱσʔλιʔ εͷ౤ೖ ύʔεͱಈతͳ σʔλม׵ ͋ΒΏΔग़ྗʹ σʔλసૹ ҆શͰ҉߸Խ͞Εͨ 
σʔλೖྗ ಠࣗͷύΠϓϥΠϯॲཧ ͷ࡞੒ 200Ҏ্ͷϓϥάΠϯ

81.

Logstash in 10 seconds • ϩάɾσʔλͷऩूɾ؅ཧ • ऩूɺύʔεɾՃ޻ɺૹग़ • ΦʔϓϯιʔεɿApache
License 2.0 • Ruby app (JRuby) !81

82.

Logstash architecture !82 Input Output Filter ? ? collect and
split alter and enrich store and visualize

83.

ઃఆ 83 input { … } filter { … }
output { … }

84.

ઃఆɿinput 84 input { file { path => “/Users/johtani/sample/*_log" start_position
=> "beginning" } }

85.

1ߦ1σʔλ 189.120.xx.xx - - [02/Dec/2014:12:18:29 +0900] "GET /manager/html HTTP/ 1.1"
404 274 "-" "Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0" 85

86.

ઃఆɿfilter 86 filter { grok { match => { "message"
=> "%{COMBINEDAPACHELOG}" } break_on_match => false } date { match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"] locale => en } geoip { source => ["clientip"] } useragent { source => "agent" target => "useragent" } }

87.

ύʔε !87 189.120.xx.xx - - [02/Dec/2014:12:18:29 +0900] "GET /manager/html HTTP/1.1"
404 274 "-" "Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0" {… "@timestamp": "2015-04-10T09:07:49.325Z", "clientip": "189.120.xx.xx", "ident": "-", "auth": "-", "timestamp": "02/Dec/2014:12:18:29 +0900", "verb": "GET", "request": "/manager/html", … "agent": "\"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/

88.

ઃఆɿfilter !88 filter { grok { match => { "message"

89.

೔෇ͷύʔε 89 {… "@timestamp": "2015-04-10T09:07:49.325Z", … "timestamp": "02/Dec/2014:12:18:29 +0900", …
} {… "@timestamp": "2014-12-02T03:18:29.000Z", … "timestamp": "02/Dec/2014:12:18:29 +0900", … }

90.


91.

IP͔ΒҢ౓ܦ౓ͳͲ෇༩ 91 "clientip": "189.120.xx.xx", "clientip": "189.120.xx.xx", "geoip": { "ip": “189.120.xxx.xxx”,
… "country_name": "Brazil", "continent_code": "SA", "region_name": "27", "city_name": "São Paulo", "latitude":

92.


93.

ϢʔβΤʔδΣϯτͷύʔε 93 "agent": "\"Mozilla/5.0 (Windows NT 5.1; rv: 5.0) Gecko/20100101
Firefox/5.0\"" "agent": "\"Mozilla/5.0 (Windows NT 5.1; rv: 5.0) Gecko/20100101 Firefox/5.0\"" "useragent": { "name": "Firefox", "os": "Windows XP", "os_name": "Windows XP", "device": "Other", "major": "5", "minor": "0"

94.

ઃఆɿoutput 94 output { elasticsearch { hosts => ["localhost"] index
=> “demo_access_log-%{+YYYY.MM.dd}” } }

95.

!95 ͞Βʹ׆༻͢Δʹ͸ʁ

96.

!96 elasticsearch-hadoop - •  D E H •  PD ecd
ER •  g D •  CH •  Ca M DMS D FERC

97.

!97

98.

!98

99.

!99

100.

ͦͷଞͷ࢖͍ํ !100

101.

!101 σʔλͷొ࿥ํ๏ • Kibanaͷαϯϓϧσʔλʢ6.4͔Βʣ • LogstashͰJDBC input • LogstashͰCSV •
FilebeatͰΞΫηεϩά • MetricbeatͰϝτϦοΫ • PacketbeatͰMySQL/PostgreSQLͷύέοτղੳ

102.

!102 Kibanaͷαϯϓϧσʔλʢ>= 6.4.0ʣ

103.

!103 ϫϯΫϦοΫͰσʔλొ࿥

104.

!104 LogstashͰJDBC Input Kibana Instances Data Store Elasticsearch Nodes Logstash
Nodes

105.

!105 JDBC Input

106.

!106 LogstashͰCSV Kibana Instances CSV  File Elasticsearch Nodes Logstash Nodes

107.

!107 CSV filter

108.

!108 FilebeatͰΞΫηεϩά Beats Log Files Kibana Instances Elasticsearch Nodes

109.

• 2ͭͷElasticsearchϓϥάΠϯΛΠϯετʔϧͯ͠ElasticsearchΛىಈ • Filebeatͷapache2ϞδϡʔϧΛ༗ޮԽ • modules.d/apache2.ymlʹΞΫηεϩάͷύεΛઃఆ • setupίϚϯυΛ࣮ߦ͔ͯ͠ΒFilebeatΛىಈ !109 FilebeatͰΞΫηεϩά

110.

MetricbeatͰϝτϦοΫ Beats Metrics Kibana Instances Elasticsearch Nodes

111.

• MetricbeatͷsystemϞδϡʔϧΛ༗ޮԽ • setupίϚϯυΛ࣮ߦ͔ͯ͠ΒFilebeatΛىಈ !111 MetricbeatͰϝτϦοΫ

112.

!112 PacketbeatͰMySQLɺPostgreSQLͷύέοτղੳ Beats Wire Data Kibana Instances Elasticsearch Nodes

113.

!113 ࢀߟจݙ • Elasticsearch - The Definitive guide ‒ http://www.elastic.co/guide/en/elasticsearch/guide/current/
index.html • ॻ੶ʢ೔ຊޠʣ ‒ σʔλ෼ੳج൫ߏஙೖ໳ ‒ Elasticsearch࣮ફΨΠυ

114.

!114 ࢀߟαΠτ • Ϣʔεέʔε • https://www.elastic.co/use-cases • DiscussʢWebϑΥʔϥϜʣ • https://discuss.elastic.co
• Elastic{ON}ͷϏσΦͱࢿྉ • https://www.elastic.co/elasticon/videos • αϙʔτϝχϡʔ • https://www.elastic.co/subscriptions

115.

Thank you! • Web : https://www.elastic.co/jp/ • Forums : https://discuss.elastic.co/
• Twitter : @johtani

様々なメトリクスやログを集めてシステム解析 - Elastic Stackの入門と活用 - / Getting started Elastic Stack for logging/metrics

様々なメトリクスやログを集めてシステム解析 - Elastic Stackの入門と活用 - / Getting started Elastic Stack for logging/metrics

Want more?

Transcript