Elastic stackで始めるログ解析入門

2016/11/19のOSC福岡2016で発表した資料です。

Transcript

1. ‹#› 2016/11/19 Evangelist at Elastic Jun Ohtani @johtani Elastic stackͰ͸͡ΊΔ ϩάղੳೖ໳
2. ‹#›
3. ΞδΣϯμ • Elastic stack঺հ • Logstash - Logऩू • Elasticsearch - ݕࡧɾղੳ • Kibana - ՄࢹԽ • ঎༻ϓϥάΠϯ঺հ 3
4. about • Me, Jun Ohtani / Technical Advocate ‒ lucene-gosenίϛολʔ ‒ ElasticSearch Server೔ຊޠ൛ͷ຋༁ ‒ http://blog.johtani.info
 • Elasticsearch, founded in 2012 ‒ Products: Elasticsearch, Logstash, Kibana, Beats 
 Marvel, Shield, Watcher, Graph
 Professional services: Support & development subscriptions ‒ Trainings 4
5. 5 ElasticελοΫ
6. ElasticελοΫʢOpen Sourceʣ 6 Kibana    
   Elasticsearch    
   
   Logstash Beats
7. ElasticελοΫ 7 Elastic Cloud  
   
   
    X-Pack Kibana    
   Elasticsearch !  "
   
   Logstash Beats +
8. Elastic stackʹΑΔσʔλ෼ੳ 8 σʔλ Import/Parse
 /Export Store/Search Visualize
9. 9 Logstash
10. Elastic stackʹΑΔσʔλ෼ੳ 10 σʔλ Import/Parse
 /Export Store/Search Visualize
11. Logstash in 10 seconds • ϩάɾσʔλͷऩूɾ؅ཧ • ऩूɺύʔεɾՃ޻ɺૹग़ • ΦʔϓϯιʔεɿApache License 2.0 • Ruby app (JRuby) 11
12. Logstash architecture 12 Input Output Filter ? ? collect and split alter and enrich store and visualize
13. ઃఆ 13 input { … } filter { … } output { … }
14. ઃఆɿinput 14 input { file { path => “/Users/johtani/sample/*_log" start_position => "beginning" } }
15. 1ߦ1σʔλ 189.120.xx.xx - - [02/Dec/2014:12:18:29 +0900] "GET /manager/html HTTP/ 1.1" 404 274 "-" "Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0" 15
16. ઃఆɿfilter 16 filter { grok { match => { "message" => "%{COMBINEDAPACHELOG}" } break_on_match => false } date { match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"] locale => en } geoip { source => ["clientip"] } useragent { source => "agent" target => "useragent" } }
17. ύʔε 17 189.120.xx.xx - - [02/Dec/2014:12:18:29 +0900] "GET /manager/html HTTP/1.1" 404 274 "-" "Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0" {… "@timestamp": "2015-04-10T09:07:49.325Z", "clientip": "189.120.xx.xx", "ident": "-", "auth": "-", "timestamp": "02/Dec/2014:12:18:29 +0900", "verb": "GET", "request": "/manager/html", … "agent": "\"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/
18. ઃఆɿfilter 18 filter { grok { match => { "message" => "%{COMBINEDAPACHELOG}" } break_on_match => false } date { match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"] locale => en } geoip { source => ["clientip"] } useragent { source => "agent" target => "useragent" } }
19. ೔෇ͷύʔε 19 {… "@timestamp": "2015-04-10T09:07:49.325Z", … "timestamp": "02/Dec/2014:12:18:29 +0900", … } {… "@timestamp": "2014-12-02T03:18:29.000Z", … "timestamp": "02/Dec/2014:12:18:29 +0900", … }
20. ઃఆɿfilter 20 filter { grok { match => { "message" => "%{COMBINEDAPACHELOG}" } break_on_match => false } date { match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"] locale => en } geoip { source => ["clientip"] } useragent { source => "agent" target => "useragent" } }
21. IP͔ΒҢ౓ܦ౓ͳͲ෇༩ 21 "clientip": "189.120.xx.xx", "clientip": "189.120.xx.xx", "geoip": { "ip": “189.120.xxx.xxx”, … "country_name": "Brazil", "continent_code": "SA", "region_name": "27", "city_name": "São Paulo", "latitude":
22. ઃఆɿfilter 22 filter { grok { match => { "message" => "%{COMBINEDAPACHELOG}" } break_on_match => false } date { match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"] locale => en } geoip { source => ["clientip"] } useragent { source => "agent" target => "useragent" } }
23. ϢʔβΤʔδΣϯτͷύʔε 23 "agent": "\"Mozilla/5.0 (Windows NT 5.1; rv: 5.0) Gecko/20100101 Firefox/5.0\"" "agent": "\"Mozilla/5.0 (Windows NT 5.1; rv: 5.0) Gecko/20100101 Firefox/5.0\"" "useragent": { "name": "Firefox", "os": "Windows XP", "os_name": "Windows XP", "device": "Other", "major": "5", "minor": "0"
24. ઃఆɿoutput 24 output { elasticsearch { hosts => ["localhost"] index => “demo_access_log-%{+YYYY.MM.dd}” } }
25. 25 Elasticsearch
26. ݕࡧͱͯ͠ͷ
 Elasticsearch
27. Elasticsearchͱ͸ʁ
28. ϑϦʔϫʔυݕࡧ 28
29. ߜΓࠐΈ 29
30. ϋΠϥΠτ 30
31. ιʔτ 31
32. ϖʔδϯά 32
33. ूܭ 33
34. αδΣετ 34
35. Elasticsearch in 10 seconds • εΩʔϚϑϦʔɺ෼ࢄυΩϡϝϯτετΞɺREST & JSON • Φʔϓϯιʔε: Apache License 2.0 • ઃఆͳ͠Ͱ؆୯ʹࢼ͢͜ͱ͕Մೳ • JavaͰ࣮૷ɻ֦ு΋༰қ 35
36. Powerful Search at Scale 36
37. ؆୯ͳCRUD
38. σʔλొ࿥ 38 curl -XPUT localhost:9200/books/book/1 -d ' { "title" : "Elasticsearch - The definitive guide", "authors" : "Clinton Gormley", "started" : "2013-02-04", "pages" : 230 }'
39. σʔλߋ৽ 39 curl -XPUT localhost:9200/books/book/1 -d ' { "title" : "Elasticsearch - The definitive guide", "authors" : [ "Clinton Gormley", "Zachary Tong" ], "started" : "2013-02-04", "pages" : 230 }'
40. σʔλ࡟আ 40 curl -X DELETE localhost:9200/books/book/1 σʔλͷऔಘ curl —X GET localhost:9200/books/book/1 curl —X GET localhost:9200/books/book/1/_source
41. ݕࡧ 41 curl -XGET localhost:9200/books/_search?q=elasticsearch { "took" : 2, "timed_out" : false, "_shards" : { "total" : 5, "successful" : 5, "failed" : 0 }, "hits" : { "total" : 1, "max_score" : 0.076713204, "hits" : [ { "_index" : “books", "_type" : “book", "_id" : "1", "_score" : 0.076713204, "_source" : { "title" : "Elasticsearch - The definitive guide", "authors" : [ "Clinton Gormley", "Zachary Tong" ], "started" : “2013-02-04", "pages" : 230 } } ]
42. ݕࡧ - Query DSL 42 curl -XGET ‘localhost:9200/books/book/_search' -d '{ "query": { "filtered" : { "query" : { "match": { "text" : { "query" : “To Be Or Not To Be", "cutoff_frequency" : 0.01 } } }, "filter" : { "range": { "price": { "gte": 20.0 "lte": 50.0
43. ෼ࢄߏ੒ɺ
 εέʔϧ
44. Basic terms • ΠϯσοΫε ‒ σʔλͷ࿦ཧతͳू߹ɻ
 RDBͷσʔλϕʔεͷΑ͏ͳ΋ͷLogical • ϨϓϦέʔγϣϯ • ಡΈࠐΈͷεέʔϥϏϦςΟ޲্ • SPOFͷղফ • γϟʔσΟϯά • ෳ਺Ϛγϯ΁σʔλΛ෼ׂ
 ॻ͖ࠐΈͷεέʔϥϏϦςΟ޲্
 σʔλϑϩʔ੍ޚ 44
45. γϟʔυͱϨϓϦΧ 45 node 1 orders products 1 4 1 2 2 3 curl -X PUT localhost:9200/orders -d '{ "settings.index.number_of_shards" : 4 "settings.index.number_of_replicas" : 1 }' curl -X PUT localhost:9200/products -d '{ "settings.index.number_of_shards" : 2 "settings.index.number_of_replicas" : 0 }'
46. γϟʔυͱϨϓϦΧ 46 node 1 orders products 1 4 1 node 2 orders products 2 2 3 4 1 2 3
47. ࣗಈతͳ෼ࢄ 47 node 1 orders products 2 1 4 1 node 2 orders products 2 2 node 3 orders products 3 4 1 3
48. શจݕࡧͱ͸ʁ
49. શจݕࡧͱ͸ʁ • શจݕࡧʢFull text searchʣͱ͸ɺίϯϐϡʔλʹ͓͍ͯɺෳ਺ͷจॻ ʢϑΝΠϧʣ͔ΒಛఆͷจࣈྻΛݕࡧ͢Δ͜ͱɻʮϑΝΠϧ໊ݕࡧʯ΍ ʮ୯ҰϑΝΠϧ಺ͷจࣈྻݕࡧʯͱҟͳΓɺʮෳ਺จॻʹ·͕ͨͬͯɺจ ॻʹؚ·ΕΔશจΛର৅ͱͨ͠ݕࡧʯͱ͍͏ҙຯͰ࢖༻͞ΕΔɻ
 ʢWikipediaΑΓʣ 49
50. ༻ޠ • ΠϯσοΫε ݕࡧΤϯδϯ͕ݕࡧʹ࢖༻͢Δσʔλͷอଘઌ • υΩϡϝϯτʢจॻʣ ‒ ݕࡧΤϯδϯʹอଘ͞Εͨσʔλ • ϑΟʔϧυ ‒ υΩϡϝϯτʹؚ·ΕΔଐੑ • ΫΤϦ ‒ ݕࡧ৚݅ɺݕࡧࣜ 50
51. ༻ޠ • εΩʔϚ ‒ υΩϡϝϯτͷߏ଄Λఆٛ͢Δ΋ͷ • λʔϜʢTermʣɺτʔΫϯʢTokenʣ ‒ ΠϯσοΫεͷΩʔʹͳΔ୯ޠʢจࣈྻʣ ‒ จষΛҰఆͷ๏ଇͰ۠੾ͬͨ୯ޠ ‒ ୯ޠ͚ͩͰͳ͘ɺ୯ޠͷҐஔͳͲ΋ؚΉ 51
52. υΩϡϝϯτͷొ࿥ 52 1 2 ΧπΦ͸αβΤͷఋ αβΤ͸ϫΧϝͷ࢞ υΩϡϝϯτͷొ࿥
53. υΩϡϝϯτͷొ࿥ 53 1 2 ΧπΦ͸αβΤͷఋ αβΤ͸ϫΧϝͷ࢞ 1 2 ΧπΦ αβΤ ͸ ͸ ͷ ͷ αβΤ ϫΧϝ ఋ ࢞ υΩϡϝϯτͷొ࿥ ୯ޠʹ෼ׂ
54. υΩϡϝϯτͷొ࿥ 54 1 2 ΧπΦ͸αβΤͷఋ αβΤ͸ϫΧϝͷ࢞ 1 2 ΧπΦ αβΤ ͸ ͸ ͷ ͷ αβΤ ϫΧϝ ఋ ࢞ ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 υΩϡϝϯτͷొ࿥ ୯ޠʹ෼ׂ ୯ޠ͔Βidͷ഑ྻ͕ Ҿ͚ΔΑ͏ʹ
55. ݕࡧ 55 ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 ݕࡧ৚݅ೖྗ ΧπΦɹαβΤ
56. ݕࡧ 56 ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ৚݅ೖྗ ݕࡧ৚݅ͷύʔε
 ݕࡧΫΤϦԽ ΧπΦɹαβΤ
57. ݕࡧ 57 ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ৚݅ೖྗ ݕࡧ৚݅ͷύʔε
 ݕࡧΫΤϦԽ ΧπΦɹαβΤ
58. ݕࡧ 58 ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ৚݅ೖྗ ݕࡧ৚݅ͷύʔε
 ݕࡧΫΤϦԽ ΧπΦɹαβΤ
59. ݕࡧ 59 ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ৚݅ೖྗ ݕࡧ৚݅ͷύʔε
 ݕࡧΫΤϦԽ ΧπΦɹαβΤ
60. ݕࡧ 60 ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ৚݅ೖྗ ݕࡧ৚݅ͷύʔε
 ݕࡧΫΤϦԽ ΧπΦɹαβΤ
61. ݕࡧ 61 ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ৚݅ೖྗ ݕࡧ৚݅ͷύʔε
 ݕࡧΫΤϦԽ ΧπΦɹαβΤ
62. ୯ޠͷ۠੾Γํ • ӳޠͷ৔߹ I am speaking Introduction Elasticsearch. 
 
 • ೔ຊޠͷ৔߹ ࢲ͸ೖ໳Elasticsearchʹ͍ͭͯ࿩͍ͯ͠Δɻ
 
 62
63. ୯ޠͷ۠੾Γํ • ӳޠͷ৔߹ I am speaking Introduction Elasticsearch. 
 
 εϖʔε͕੾Ε໨ͱΘ͔Δ • ೔ຊޠͷ৔߹ ࢲ͸ೖ໳Elasticsearchʹ͍ͭͯ࿩͍ͯ͠Δɻ
 Ͳ͜Ͱ۠੾Ε͹Α͍ʁ 63
64. N-Gramͱܗଶૉղੳ • సஔΠϯσοΫεͷΩʔͷ࡞Γํ ‒ ೔ຊޠ͸୯ޠͷ੾Ε໨͕Θ͔Βͳ͍ͷͰɺసஔΠϯσοΫεͷΩʔ͸ ओʹ࣍ͷ̎ͭͷख๏Ͱ࡞੒ • N-Gram ‒ NจࣈͣͭจষΛ۠੾Δ • ܗଶૉղੳ ‒ ࣙॻͳͲΛ༻͍ͯҙຯͷ͋Δ୯ޠͰ۠੾Δ 64
65. ܗଶૉղੳ • ϝϦοτɿ ‒ ҙຯͷ͋Δ୯ޠͷ੾Ε໨
 ඼ࢺ৘ใΛݩʹ௥Ճॲཧ͕Մೳʢޠװม׵ͳͲʣ • σϝϦοτɿ ‒ ৽ޠʢະ஌ޠʣʹऑ͍→ࣙॻϕʔεͷ৔߹ɺࣙॻʹͳ͍୯ޠ͸ݕग़ෆ ೳɻ 65 ΧπΦ͸αβΤͷఋ ΧπΦ ͸ ͷ αβΤ ఋ
66. N-Gram • ϝϦοτɿ ‒ ະ஌ޠʹରԠՄೳ • σϝϦοτɿ ‒ ΠϯσοΫεංେԽ ‒ ඼ࢺ৘ใʹجͮ͘ॲཧ͕ෆՄೳ 66 ΧπΦ͸αβΤͷఋ Χπ πΦ Φ͸ ͸α αβ βΤ Τͷ ͷఋ
67. ͦͷଞͷػೳ
68. elasticsearch ͞·͟·ͳܗࣜͷσʔλͰ GeoݕࡧՄೳ
 
 Ң౓ܦ౓ɺGeoHashɺ GeoShape… GEO
69. Percolator • υΩϡϝϯτͰ͸ͳ͘ΫΤϦΛొ࿥ • _percolate APIʹυΩϡϝϯτΛૹ৴ • ొ࿥͞ΕͨΫΤϦʹϚονͨ͠৔߹ɺ
 Ϛονͨ͠ΫΤϦΛฦ͢ 69
70. Snapshot/Restore • 1.0͔Βಋೖ • ΠϯσοΫε୯ҐͰSnapshot/RestoreՄೳ • อଘઌʢRepositoryʣ ‒ Shard FSɺS3ɺHDFSɺAzure… 70
71. Ecosystem • Plugins ‒ ϓϥάΠϯʹΑΔػೳͷ௥Ճ • ΫϥΠΞϯτϥΠϒϥϦ • Ruby, python, php, perl, javascript, .NET • Scala, clojure, go • Hadoop integration 71
72. Elasticsearch - The Definitive guide
 
 http://www.elastic.co/guide/en/ elasticsearch/guide/current/index.html 72 ৄ͘͠஌Γ͍ͨํ͸
73. ղੳͱͯ͠ͷElasticsearch
74. aggregation
75. Aggregationͱ͸ • 1.0͔Βಋೖ • FacetΑΓ΋ڧྗͳूܭͳͲ͕Մೳ • ֊૚తͳूܭɺάϧʔϓԽ
 ಈతͳूܭɺάϧʔϓԽ • େ͖͘2छྨ • BucketɹυΩϡϝϯτΛ஋͝ͱʹ݁ՌΛάϧʔϐϯά • Metricɹ υΩϡϝϯτͷ࣋ͭ஋Λूܭ 75
76. ྫɿݴޠ͓Αͼ஍Ҭͷूܭ 76 curl -XGET twitter-2014.08.22/_search -d ' { "aggs": { "lang": { "terms": {"field": "lang" }, "aggs": { "place": { "terms": { "field": “place.full_name", "size": 10 } } } } } }
77. ྫɿݴޠ͓Αͼ஍Ҭͷूܭ 77 "aggregations": { "lang": { "buckets": [{…}, { "key": "ja", "doc_count": 980145, "place": { "buckets": [ { "key": "ژ౎ࢢ෬ݟ۠, ژ౎", "doc_count":252 }, { "key": "ઍ୅ా۠, ౦ژ", "doc_count": 39 },…
78. 78 KibanaͰՄࢹԽ
79. Kibana 4 • ElasticsearchͷσʔλΛՄࢹԽ • Node.js server & JavaScript • Apache License 2.0 • Elastic Stackͷ૭ͷ໾ׂ • ༷ʑͳGUIΛPluginͱ͍ͯެ։ • MarvelɺSenseɺTimelionͳͲ 79
80. Kibana 4 80
81. σϞ for Kibana5 Access Log 81
82. Combining Search and Analytics 82
83. ͦͷଞͷ ElasticελοΫ
84. ܰྔσʔλγούʔ 84 Beats
85. Capture the Packet Packetbeat
86. Capture the Packet Packetbeat
87. Welcome to 1998 winlogbeat
88. Now winlogbeat
89. elasticsearch-hadoop 89 - •  D E H •  PD ecd ER •  g D •  CH •  Ca M DMS D FERC
91. X-Pack 5.0: Extending the Elastic Stack
92. Security
93. X-Pack : Securityͷಛ௃ • User Authentication ‒ LDAP/Active Directory/ϑΝΠϧϕʔε • Authorization ‒ ϩʔϧϕʔεͷΞΫηείϯτϩʔϧ ‒ ΠϯσοΫε͝ͱɺΞΫγϣϯ͝ͱͷઃఆ͕Մೳ ‒ υΩϡϝϯτɾϑΟʔϧυ͝ͱͷઃఆ΋Մೳʹ • ηΩϡΞͳ௨৴ ‒ ElasticsearchϊʔυؒͷSSL/TLSɺIPϑΟϧλϦϯά • ؂ࠪϩά 93
94. ΨΠυͳͲ • ϓϩμΫτϖʔδ ‒ https://www.elastic.co/jp/products/x-pack/security • ΨΠυ ‒ https://www.elastic.co/guide/en/x-pack/current/xpack-security.html 94
95. Alerting
96. X-Pack : Alertingͷಛ௃ • ΫΤϦʹΑΔWatch ‒ ElasticsearchͷΫΤϦΛར༻ͯ͠σʔλͷ؂ࢹ • ৚݅ͷઃఆ ‒ ΞΫγϣϯΛ࣮ߦ͢Δ͔Ͳ͏͔ͷઃఆ • εέδϡʔϧ ‒ ΫΤϦΛ࣮ߦ͠ɺ৚݅ΛνΣοΫ͢Δස౓ͷࢦఆ • ΞΫγϣϯͷఆٛ ‒ ϝʔϧͷૹ৴ɺଞγεςϜ΁ͷσʔλૹ৴ͳͲͷಈ࡞Λઃఆ • ཤྺͷอଘ 96
97. ΨΠυͳͲ • ϓϩμΫτϖʔδ ‒ https://www.elastic.co/jp/products/x-pack/alerting
 
 • ΨΠυ ‒ https://www.elastic.co/guide/en/x-pack/current/xpack-alerting.html 97
98. Graph
99. Graphͷಛ௃ • σʔλؒͷͭͳ͕ΓΛ୳ࡧ͢ΔϓϥάΠϯ • KibanaϓϥάΠϯʹΑΓGUIΛར༻ͯ͠୳ࡧՄೳ 99
100. ΨΠυͳͲ • ϓϩμΫτϖʔδ ‒ https://www.elastic.co/jp/products/x-pack/graph
 
 • ΨΠυ ‒ https://www.elastic.co/guide/en/x-pack/current/xpack-graph.html 100
101. Cloud
102. Elastic Cloud • Elasticsearch as a Service • Elasticsearchͷػೳ͕ར༻Մೳ • ϓϥάΠϯͷར༻΋Մೳ • ΞοϓάϨʔυ΋؆୯ • ΧελϜࣙॻɺϓϥάΠϯ΋ར༻Մೳ • 14೔ؒͷ͓ࢼ͠ظؒ͋Γ • SLAϕʔεͷαϒεΫϦϓγϣϯ+঎༻ϓϥάΠϯ
 ͕ར༻ՄೳͳαʔϏε΋ 102 Elastic Cloud
104. ࢀߟจݙ • Elasticsearch - The Definitive guide ‒ http://www.elastic.co/guide/en/elasticsearch/guide/current/index.html • ॻ੶ʢ೔ຊޠʣ ‒ ElasticSearchServer೔ຊޠ൛
 αʔό/ΠϯϑϥΤϯδχΞ
 ɹཆ੒ಡຊɹϩάऩू 104
105. ࢀߟαΠτ • Ϣʔεέʔε • https://www.elastic.co/use-cases • DiscussʢWebϑΥʔϥϜʣ • https://discuss.elastic.co • Elastic{ON}ͷϏσΦͱࢿྉ • https://www.elastic.co/elasticon/videos • αϙʔτϝχϡʔ • https://www.elastic.co/subscriptions 105
106. Thanks for listening! Q & A We’re hiring! https://www.elastic.co/about/careers/ We’re helping! https://www.elastic.co/subscriptions http://training.elastic.co