Elastic stackではじめる ログ解析入門

Transcript

1. ‹#› 2017/01/27 Evangelist at Elastic Jun Ohtani @johtani Elastic stackͰ͸͡ΊΔ

   ϩάղੳೖ໳

2. ‹#›

3. ΞδΣϯμ • Elastic stack঺հ • Logstash - Logऩू • Elasticsearch

   - ݕࡧɾղੳ • Kibana - ՄࢹԽ • ঎༻ϓϥάΠϯ঺հ 3

4. about • Me, Jun Ohtani / Technical Advocate ‒ lucene-gosenίϛολʔ

   ‒ ElasticSearch Server೔ຊޠ൛ͷ຋༁ ‒ http://blog.johtani.info
 • Elasticsearch, founded in 2012 ‒ Products: Elasticsearch, Logstash, Kibana, Beats 
 Marvel, Shield, Watcher, Graph
 Professional services: Support & development subscriptions ‒ Trainings 4

5. 5 ElasticελοΫ

6. ElasticελοΫʢOpen Sourceʣ 6 Kibana    
   Elasticsearch

      
   
   Logstash Beats

7. ElasticελοΫ 7 Elastic Cloud  
   
   
   

   X-Pack Kibana    
   Elasticsearch !  "
   
   Logstash Beats +

8. Elastic stackʹΑΔσʔλ෼ੳ 8 σʔλ Import/Parse
 /Export Store/Search Visualize

9. 9 Logstash

10. Elastic stackʹΑΔσʔλ෼ੳ 10 σʔλ Import/Parse
 /Export Store/Search Visualize

11. Logstash in 10 seconds • ϩάɾσʔλͷऩूɾ؅ཧ • ऩूɺύʔεɾՃ޻ɺૹग़ • ΦʔϓϯιʔεɿApache

    License 2.0 • Ruby app (JRuby) 11

12. Logstash architecture 12 Input Output Filter ? ? collect and

    split alter and enrich store and visualize

13. ઃఆ 13 input { … } filter { … }

    output { … }

14. ઃఆɿinput 14 input { file { path => “/Users/johtani/sample/*_log" start_position

    => "beginning" } }

15. 1ߦ1σʔλ 189.120.xx.xx - - [02/Dec/2014:12:18:29 +0900] "GET /manager/html HTTP/ 1.1"

    404 274 "-" "Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0" 15

16. ઃఆɿfilter 16 filter { grok { match => { "message"

    => "%{COMBINEDAPACHELOG}" } break_on_match => false } date { match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"] locale => en } geoip { source => ["clientip"] } useragent { source => "agent" target => "useragent" } }

17. ύʔε 17 189.120.xx.xx - - [02/Dec/2014:12:18:29 +0900] "GET /manager/html HTTP/1.1"

    404 274 "-" "Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0" {… "@timestamp": "2015-04-10T09:07:49.325Z", "clientip": "189.120.xx.xx", "ident": "-", "auth": "-", "timestamp": "02/Dec/2014:12:18:29 +0900", "verb": "GET", "request": "/manager/html", … "agent": "\"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/

18. ઃఆɿfilter 18 filter { grok { match => { "message"

    => "%{COMBINEDAPACHELOG}" } break_on_match => false } date { match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"] locale => en } geoip { source => ["clientip"] } useragent { source => "agent" target => "useragent" } }

19. ೔෇ͷύʔε 19 {… "@timestamp": "2015-04-10T09:07:49.325Z", … "timestamp": "02/Dec/2014:12:18:29 +0900", …

    } {… "@timestamp": "2014-12-02T03:18:29.000Z", … "timestamp": "02/Dec/2014:12:18:29 +0900", … }

20. ઃఆɿfilter 20 filter { grok { match => { "message"

    => "%{COMBINEDAPACHELOG}" } break_on_match => false } date { match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"] locale => en } geoip { source => ["clientip"] } useragent { source => "agent" target => "useragent" } }

21. IP͔ΒҢ౓ܦ౓ͳͲ෇༩ 21 "clientip": "189.120.xx.xx", "clientip": "189.120.xx.xx", "geoip": { "ip": “189.120.xxx.xxx”,

    … "country_name": "Brazil", "continent_code": "SA", "region_name": "27", "city_name": "São Paulo", "latitude":

22. ઃఆɿfilter 22 filter { grok { match => { "message"

    => "%{COMBINEDAPACHELOG}" } break_on_match => false } date { match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"] locale => en } geoip { source => ["clientip"] } useragent { source => "agent" target => "useragent" } }

23. ϢʔβΤʔδΣϯτͷύʔε 23 "agent": "\"Mozilla/5.0 (Windows NT 5.1; rv: 5.0) Gecko/20100101

    Firefox/5.0\"" "agent": "\"Mozilla/5.0 (Windows NT 5.1; rv: 5.0) Gecko/20100101 Firefox/5.0\"" "useragent": { "name": "Firefox", "os": "Windows XP", "os_name": "Windows XP", "device": "Other", "major": "5", "minor": "0"

24. ઃఆɿoutput 24 output { elasticsearch { hosts => ["localhost"] index

    => “demo_access_log-%{+YYYY.MM.dd}” } }

25. 25 Elasticsearch

26. ݕࡧͱͯ͠ͷ
 Elasticsearch

27. Elasticsearchͱ͸ʁ

28. ϑϦʔϫʔυݕࡧ 28

29. ߜΓࠐΈ 29

30. ϋΠϥΠτ 30

31. ιʔτ 31

32. ϖʔδϯά 32

33. ूܭ 33

34. αδΣετ 34

35. Elasticsearch in 10 seconds • εΩʔϚϑϦʔɺ෼ࢄυΩϡϝϯτετΞɺREST & JSON • Φʔϓϯιʔε:

    Apache License 2.0 • ઃఆͳ͠Ͱ؆୯ʹࢼ͢͜ͱ͕Մೳ • JavaͰ࣮૷ɻ֦ு΋༰қ 35

36. Powerful Search at Scale 36

37. ؆୯ͳCRUD

38. σʔλొ࿥ 38 curl -XPUT localhost:9200/books/book/1 -d ' { "title" :

    "Elasticsearch - The definitive guide", "authors" : "Clinton Gormley", "started" : "2013-02-04", "pages" : 230 }'

39. σʔλߋ৽ 39 curl -XPUT localhost:9200/books/book/1 -d ' { "title" :

    "Elasticsearch - The definitive guide", "authors" : [ "Clinton Gormley", "Zachary Tong" ], "started" : "2013-02-04", "pages" : 230 }'

40. σʔλ࡟আ 40 curl -X DELETE localhost:9200/books/book/1 σʔλͷऔಘ curl —X GET

    localhost:9200/books/book/1 curl —X GET localhost:9200/books/book/1/_source

41. ݕࡧ 41 curl -XGET localhost:9200/books/_search?q=elasticsearch { "took" : 2, "timed_out"

    : false, "_shards" : { "total" : 5, "successful" : 5, "failed" : 0 }, "hits" : { "total" : 1, "max_score" : 0.076713204, "hits" : [ { "_index" : “books", "_type" : “book", "_id" : "1", "_score" : 0.076713204, "_source" : { "title" : "Elasticsearch - The definitive guide", "authors" : [ "Clinton Gormley", "Zachary Tong" ], "started" : “2013-02-04", "pages" : 230 } } ]

42. ݕࡧ - Query DSL 42 curl -XGET ‘localhost:9200/books/book/_search' -d '{

    "query": { "filtered" : { "query" : { "match": { "text" : { "query" : “To Be Or Not To Be", "cutoff_frequency" : 0.01 } } }, "filter" : { "range": { "price": { "gte": 20.0 "lte": 50.0

43. ෼ࢄߏ੒ɺ
 εέʔϧ

44. Basic terms • ΠϯσοΫε ‒ σʔλͷ࿦ཧతͳू߹ɻ
 RDBͷσʔλϕʔεͷΑ͏ͳ΋ͷLogical • ϨϓϦέʔγϣϯ •

    ಡΈࠐΈͷεέʔϥϏϦςΟ޲্ • SPOFͷղফ • γϟʔσΟϯά • ෳ਺Ϛγϯ΁σʔλΛ෼ׂ
 ॻ͖ࠐΈͷεέʔϥϏϦςΟ޲্
 σʔλϑϩʔ੍ޚ 44

45. γϟʔυͱϨϓϦΧ 45 node 1 orders products 1 4 1 2

    2 3 curl -X PUT localhost:9200/orders -d '{ "settings.index.number_of_shards" : 4 "settings.index.number_of_replicas" : 1 }' curl -X PUT localhost:9200/products -d '{ "settings.index.number_of_shards" : 2 "settings.index.number_of_replicas" : 0 }'

46. γϟʔυͱϨϓϦΧ 46 node 1 orders products 1 4 1 node

    2 orders products 2 2 3 4 1 2 3

47. ࣗಈతͳ෼ࢄ 47 node 1 orders products 2 1 4 1

    node 2 orders products 2 2 node 3 orders products 3 4 1 3

48. શจݕࡧͱ͸ʁ

49. શจݕࡧͱ͸ʁ • શจݕࡧʢFull text searchʣͱ͸ɺίϯϐϡʔλʹ͓͍ͯɺෳ਺ͷจॻ ʢϑΝΠϧʣ͔ΒಛఆͷจࣈྻΛݕࡧ͢Δ͜ͱɻʮϑΝΠϧ໊ݕࡧʯ΍ ʮ୯ҰϑΝΠϧ಺ͷจࣈྻݕࡧʯͱҟͳΓɺʮෳ਺จॻʹ·͕ͨͬͯɺจ ॻʹؚ·ΕΔશจΛର৅ͱͨ͠ݕࡧʯͱ͍͏ҙຯͰ࢖༻͞ΕΔɻ
 ʢWikipediaΑΓʣ 49

50. ༻ޠ • ΠϯσοΫε ݕࡧΤϯδϯ͕ݕࡧʹ࢖༻͢Δσʔλͷอଘઌ • υΩϡϝϯτʢจॻʣ ‒ ݕࡧΤϯδϯʹอଘ͞Εͨσʔλ • ϑΟʔϧυ

    ‒ υΩϡϝϯτʹؚ·ΕΔଐੑ • ΫΤϦ ‒ ݕࡧ৚݅ɺݕࡧࣜ 50

51. ༻ޠ • εΩʔϚ ‒ υΩϡϝϯτͷߏ଄Λఆٛ͢Δ΋ͷ • λʔϜʢTermʣɺτʔΫϯʢTokenʣ ‒ ΠϯσοΫεͷΩʔʹͳΔ୯ޠʢจࣈྻʣ ‒

    จষΛҰఆͷ๏ଇͰ۠੾ͬͨ୯ޠ ‒ ୯ޠ͚ͩͰͳ͘ɺ୯ޠͷҐஔͳͲ΋ؚΉ 51

52. υΩϡϝϯτͷొ࿥ 52 1 2 ΧπΦ͸αβΤͷఋ αβΤ͸ϫΧϝͷ࢞ υΩϡϝϯτͷొ࿥

53. υΩϡϝϯτͷొ࿥ 53 1 2 ΧπΦ͸αβΤͷఋ αβΤ͸ϫΧϝͷ࢞ 1 2 ΧπΦ αβΤ

    ͸ ͸ ͷ ͷ αβΤ ϫΧϝ ఋ ࢞ υΩϡϝϯτͷొ࿥ ୯ޠʹ෼ׂ

54. υΩϡϝϯτͷొ࿥ 54 1 2 ΧπΦ͸αβΤͷఋ αβΤ͸ϫΧϝͷ࢞ 1 2 ΧπΦ αβΤ

    ͸ ͸ ͷ ͷ αβΤ ϫΧϝ ఋ ࢞ ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 υΩϡϝϯτͷొ࿥ ୯ޠʹ෼ׂ ୯ޠ͔Βidͷ഑ྻ͕ Ҿ͚ΔΑ͏ʹ

55. ݕࡧ 55 ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞

    ϫΧϝ 2 1 2 1 2 1 ఋ 2 ݕࡧ৚݅ೖྗ ΧπΦɹαβΤ

56. ݕࡧ 56 ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞

    ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ৚݅ೖྗ ݕࡧ৚݅ͷύʔε
 ݕࡧΫΤϦԽ ΧπΦɹαβΤ

57. ݕࡧ 57 ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞

    ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ৚݅ೖྗ ݕࡧ৚݅ͷύʔε
 ݕࡧΫΤϦԽ ΧπΦɹαβΤ

58. ݕࡧ 58 ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞

    ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ৚݅ೖྗ ݕࡧ৚݅ͷύʔε
 ݕࡧΫΤϦԽ ΧπΦɹαβΤ

59. ݕࡧ 59 ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞

    ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ৚݅ೖྗ ݕࡧ৚݅ͷύʔε
 ݕࡧΫΤϦԽ ΧπΦɹαβΤ

60. ݕࡧ 60 ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞

    ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ৚݅ೖྗ ݕࡧ৚݅ͷύʔε
 ݕࡧΫΤϦԽ ΧπΦɹαβΤ

61. ݕࡧ 61 ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞

    ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ৚݅ೖྗ ݕࡧ৚݅ͷύʔε
 ݕࡧΫΤϦԽ ΧπΦɹαβΤ

62. ୯ޠͷ۠੾Γํ • ӳޠͷ৔߹ I am speaking Introduction Elasticsearch. 
 


    • ೔ຊޠͷ৔߹ ࢲ͸ೖ໳Elasticsearchʹ͍ͭͯ࿩͍ͯ͠Δɻ
 
 62

63. ୯ޠͷ۠੾Γํ • ӳޠͷ৔߹ I am speaking Introduction Elasticsearch. 
 


    εϖʔε͕੾Ε໨ͱΘ͔Δ • ೔ຊޠͷ৔߹ ࢲ͸ೖ໳Elasticsearchʹ͍ͭͯ࿩͍ͯ͠Δɻ
 Ͳ͜Ͱ۠੾Ε͹Α͍ʁ 63

64. N-Gramͱܗଶૉղੳ • సஔΠϯσοΫεͷΩʔͷ࡞Γํ ‒ ೔ຊޠ͸୯ޠͷ੾Ε໨͕Θ͔Βͳ͍ͷͰɺసஔΠϯσοΫεͷΩʔ͸ ओʹ࣍ͷ̎ͭͷख๏Ͱ࡞੒ • N-Gram ‒ NจࣈͣͭจষΛ۠੾Δ

    • ܗଶૉղੳ ‒ ࣙॻͳͲΛ༻͍ͯҙຯͷ͋Δ୯ޠͰ۠੾Δ 64

65. ܗଶૉղੳ • ϝϦοτɿ ‒ ҙຯͷ͋Δ୯ޠͷ੾Ε໨
 ඼ࢺ৘ใΛݩʹ௥Ճॲཧ͕Մೳʢޠװม׵ͳͲʣ • σϝϦοτɿ ‒ ৽ޠʢະ஌ޠʣʹऑ͍→ࣙॻϕʔεͷ৔߹ɺࣙॻʹͳ͍୯ޠ͸ݕग़ෆ

    ೳɻ 65 ΧπΦ͸αβΤͷఋ ΧπΦ ͸ ͷ αβΤ ఋ

66. N-Gram • ϝϦοτɿ ‒ ະ஌ޠʹରԠՄೳ • σϝϦοτɿ ‒ ΠϯσοΫεංେԽ ‒

    ඼ࢺ৘ใʹجͮ͘ॲཧ͕ෆՄೳ 66 ΧπΦ͸αβΤͷఋ Χπ πΦ Φ͸ ͸α αβ βΤ Τͷ ͷఋ

67. ͦͷଞͷػೳ

68. elasticsearch ͞·͟·ͳܗࣜͷσʔλͰ GeoݕࡧՄೳ
 
 Ң౓ܦ౓ɺGeoHashɺ GeoShape… GEO

69. Percolator • υΩϡϝϯτͰ͸ͳ͘ΫΤϦΛొ࿥ • _percolate APIʹυΩϡϝϯτΛૹ৴ • ొ࿥͞ΕͨΫΤϦʹϚονͨ͠৔߹ɺ
 Ϛονͨ͠ΫΤϦΛฦ͢ 69

70. Snapshot/Restore • 1.0͔Βಋೖ • ΠϯσοΫε୯ҐͰSnapshot/RestoreՄೳ • อଘઌʢRepositoryʣ ‒ Shard FSɺS3ɺHDFSɺAzure…

    70

71. Ecosystem • Plugins ‒ ϓϥάΠϯʹΑΔػೳͷ௥Ճ • ΫϥΠΞϯτϥΠϒϥϦ • Ruby, python,

    php, perl, javascript, .NET • Scala, clojure, go • Hadoop integration 71

72. Elasticsearch - The Definitive guide
 
 http://www.elastic.co/guide/en/ elasticsearch/guide/current/index.html 72 ৄ͘͠஌Γ͍ͨํ͸

73. ղੳͱͯ͠ͷElasticsearch

74. aggregation

75. Aggregationͱ͸ • 1.0͔Βಋೖ • FacetΑΓ΋ڧྗͳूܭͳͲ͕Մೳ • ֊૚తͳूܭɺάϧʔϓԽ
 ಈతͳूܭɺάϧʔϓԽ • େ͖͘2छྨ

    • BucketɹυΩϡϝϯτΛ஋͝ͱʹ݁ՌΛάϧʔϐϯά • Metricɹ υΩϡϝϯτͷ࣋ͭ஋Λूܭ 75

76. ྫɿݴޠ͓Αͼ஍Ҭͷूܭ 76 curl -XGET twitter-2014.08.22/_search -d ' { "aggs": {

    "lang": { "terms": {"field": "lang" }, "aggs": { "place": { "terms": { "field": “place.full_name", "size": 10 } } } } } }

77. ྫɿݴޠ͓Αͼ஍Ҭͷूܭ 77 "aggregations": { "lang": { "buckets": [{…}, { "key":

    "ja", "doc_count": 980145, "place": { "buckets": [ { "key": "ژ౎ࢢ෬ݟ۠, ژ౎", "doc_count":252 }, { "key": "ઍ୅ా۠, ౦ژ", "doc_count": 39 },…

78. 78 KibanaͰՄࢹԽ

79. Kibana 5 • ElasticsearchͷσʔλΛՄࢹԽ • Node.js server & JavaScript •

    Apache License 2.0 • Elastic Stackͷ૭ͷ໾ׂ • ༷ʑͳGUIΛPluginͱ͍ͯެ։ • MarvelɺSenseɺTimelionͳͲ 79

80. Kibana 5 80

81. σϞ for Kibana5 Access Log 81

82. Combining Search and Analytics 82

83. ͦͷଞͷ ElasticελοΫ

84. ܰྔσʔλγούʔ 84 Beats

85. Capture the Packet Packetbeat

86. Capture the Packet Packetbeat

87. Welcome to 1998 winlogbeat

88. Now winlogbeat

89. elasticsearch-hadoop 89 - •  D E H •  PD ecd

    ER •  g D •  CH •  Ca M DMS D FERC
90. None

91. X-Pack 5.0: Extending the Elastic Stack

92. Security

93. X-Pack : Securityͷಛ௃ • User Authentication ‒ LDAP/Active Directory/ϑΝΠϧϕʔε •

    Authorization ‒ ϩʔϧϕʔεͷΞΫηείϯτϩʔϧ ‒ ΠϯσοΫε͝ͱɺΞΫγϣϯ͝ͱͷઃఆ͕Մೳ ‒ υΩϡϝϯτɾϑΟʔϧυ͝ͱͷઃఆ΋Մೳʹ • ηΩϡΞͳ௨৴ ‒ ElasticsearchϊʔυؒͷSSL/TLSɺIPϑΟϧλϦϯά • ؂ࠪϩά 93

94. ΨΠυͳͲ • ϓϩμΫτϖʔδ ‒ https://www.elastic.co/jp/products/x-pack/security • ΨΠυ ‒ https://www.elastic.co/guide/en/x-pack/current/xpack-security.html 94

95. Alerting

96. X-Pack : Alertingͷಛ௃ • ΫΤϦʹΑΔWatch ‒ ElasticsearchͷΫΤϦΛར༻ͯ͠σʔλͷ؂ࢹ • ৚݅ͷઃఆ ‒

    ΞΫγϣϯΛ࣮ߦ͢Δ͔Ͳ͏͔ͷઃఆ • εέδϡʔϧ ‒ ΫΤϦΛ࣮ߦ͠ɺ৚݅ΛνΣοΫ͢Δස౓ͷࢦఆ • ΞΫγϣϯͷఆٛ ‒ ϝʔϧͷૹ৴ɺଞγεςϜ΁ͷσʔλૹ৴ͳͲͷಈ࡞Λઃఆ • ཤྺͷอଘ 96

97. ΨΠυͳͲ • ϓϩμΫτϖʔδ ‒ https://www.elastic.co/jp/products/x-pack/alerting
 
 • ΨΠυ ‒ https://www.elastic.co/guide/en/x-pack/current/xpack-alerting.html

    97

98. Graph

99. Graphͷಛ௃ • σʔλؒͷͭͳ͕ΓΛ୳ࡧ͢ΔϓϥάΠϯ • KibanaϓϥάΠϯʹΑΓGUIΛར༻ͯ͠୳ࡧՄೳ 99

100. ΨΠυͳͲ • ϓϩμΫτϖʔδ ‒ https://www.elastic.co/jp/products/x-pack/graph
 
 • ΨΠυ ‒ https://www.elastic.co/guide/en/x-pack/current/xpack-graph.html

     100

101. Cloud

102. Elastic Cloud • Elasticsearch as a Service • Elasticsearchͷػೳ͕ར༻Մೳ •

     ϓϥάΠϯͷར༻΋Մೳ • ΞοϓάϨʔυ΋؆୯ • ΧελϜࣙॻɺϓϥάΠϯ΋ར༻Մೳ • 14೔ؒͷ͓ࢼ͠ظؒ͋Γ • SLAϕʔεͷαϒεΫϦϓγϣϯ+঎༻ϓϥάΠϯ
 ͕ར༻ՄೳͳαʔϏε΋ 102 Elastic Cloud

103. 103

104. ࢀߟจݙ • Elasticsearch - The Definitive guide ‒ http://www.elastic.co/guide/en/elasticsearch/guide/current/index.html •

     ॻ੶ʢ೔ຊޠʣ ‒ ElasticSearchServer೔ຊޠ൛
 αʔό/ΠϯϑϥΤϯδχΞ
 ɹཆ੒ಡຊɹϩάऩू 104

105. ࢀߟαΠτ • Ϣʔεέʔε • https://www.elastic.co/use-cases • DiscussʢWebϑΥʔϥϜʣ • https://discuss.elastic.co •

     Elastic{ON}ͷϏσΦͱࢿྉ • https://www.elastic.co/elasticon/videos • αϙʔτϝχϡʔ • https://www.elastic.co/subscriptions 105

106. Thanks for listening! Q & A We’re hiring! https://www.elastic.co/about/careers/ We’re

     helping! https://www.elastic.co/subscriptions http://training.elastic.co