Pro Yearly is on sale from $80 to $50! »

様々なメトリクスやログを集めてシステム解析 
- Elastic Stackの入門と活用 - / Intro Elastic Stack

657aeeff3fc467567dacebf8a1ea0b23?s=47 Jun Ohtani
July 11, 2018
85

様々なメトリクスやログを集めてシステム解析 
- Elastic Stackの入門と活用 - / Intro Elastic Stack

OSC Hokkaido 2018での発表資料です。

657aeeff3fc467567dacebf8a1ea0b23?s=128

Jun Ohtani

July 11, 2018
Tweet

Transcript

  1. !1 2018/07/06 Developer Advocate at Elastic Jun Ohtani @johtani ༷ʑͳϝτϦΫε΍ϩάΛूΊͯγεςϜղੳ

    
 - Elastic Stackͷೖ໳ͱ׆༻ -
  2. !2

  3. !3 ΞδΣϯμ • ϝτϦΫεʗϩάͱ͸ʁ • γεςϜϝτϦΫεղੳɺϩάղੳΛࢼ͠ʹ΍ͬͯΈΑ͏ • Beats - Elasticsearch

    - KibanaͰղੳ • ຊ֨తʹղੳΛ΍Δʹ͸ʁ • LogstashͰϩά΍ϝτϦΫεΛதܧɾू໿ • ͞Βʹ৭ʑࢼͯ͠ΈΔʹ͸ʁ
  4. !4 about • Me, Jun Ohtani / Developer Advocate ‒

    lucene-gosenίϛολʔ ‒ σʔλ෼ੳج൫ߏஙೖ໳ ڞஶ ‒ http://blog.johtani.info
 • Elasticsearch, founded in 2012 ‒ Products: Elasticsearch, Logstash, Kibana, Beats 
 Elastic APM, 
 X-Pack, Elastic Cloud, Swiftype 
 Professional services: Support & development subscriptions
 Trainings, Consulting, SaaS
  5. !5 ͲΜͳϝτϦΫεɺ
 ϩάΛूΊ͍ͯ·͔͢ʁ

  6. !6 ϝτϦΫε • CPUɺϝϞϦ࢖༻཰ɺσΟεΫ࢖༻཰ • ΞΫηε਺ɺωοτϫʔΫసૹྔ • Ԡ౴࣌ؒ • ίωΫγϣϯ਺

    • τϥϯβΫγϣϯ਺ɺച্ • ίϯςφͷ্ͷ֤छϝτϦΫε
  7. !7 ϩά • ೝূϩά • γεςϜϩά • ΞϓϦέʔγϣϯϩά • Slow

    log • ΞΫηεϩά • ίϯςφͷதͷϩά
  8. !8 Ͱ͖Ε͹ϩάͱϝτϦΫεΛ
 ·ͱΊͯ1ͭͷը໘Ͱ
 ݟ͍ͨͰ͢ΑͶʁ

  9. !9 Elastic Stack

  10. 10 Elastic Stack 100% Φʔϓϯιʔε ʮΤϯλʔϓϥΠζ൛ʯ͸ແ͠ όʔδϣϯ 5.0Ͱ׬શ౷Ұ

  11. !11 X-Pack ؆୯ʹΠϯετʔϧ Elastic StackΛ֦ு αϒεΫϦϓγϣϯʹؚ·ΕΔ Security Alerting Monitoring Reporting

    Graph Machine Learning
  12. ఆܕͷϝτϦΫε/ϩάղੳΛ Elastic StackͰ

  13. !13 ϝτϦΫεɾϩάͷ෼ੳʢ؆қ൛ʣ σʔλ Import Parse/
 Store/Search Visualize

  14. !14

  15. 15 Beats ܰྔσʔλγούʔ ιʔε͔ΒσʔλΛసૹ సૹ͠Elasticsearchʹू໿ ม׵ͱύʔεͷͨΊ Logstashʹసૹ Elastic Cloudʹసૹ Libbeat:

    ΧελϜbeatsͷͨ ΊͷAPIϑϨʔϜϫʔΫ 30Ҏ্ͷίϛϡχςΟbeats
  16. The Beats family Heartbeat Uptime monitoring Filebeat Log files Winlogbeat

    Windows Event Logs Packetbeat Network data +40 community Beats Metricbeat Metrics Auditbeat Audit data
  17. Collect system and application metrics Metricbeat

  18. lots of modules Metricbeat

  19. tail log from file Filebeat

  20. many modules Filebeat

  21. Capture the Packet Packetbeat

  22. Capture the Packet Packetbeat

  23. Welcome to 1998 winlogbeat

  24. Now winlogbeat

  25. !25 • Kubernetes module in Metricbeat ‒ CPU, memory, ωοτϫʔΫసૹྔͳͲ

    • add_docker_metadataϓϩηοα ‒ Container ID, name, image, labels • add_kubernetes_metadataϓϩηοα ‒ Pod name, pod namespace, container name, pod labels Beats <3 ίϯςφ Docker΍KubernetesͰͷσϓϩΠΛ؆୯ʹ
  26. !26

  27. 27 Elasticsearch Heart of the Elastic Stack ෼ࢄܕɺεέʔϥϒϧ ߴՄ༻ੑ Ϛϧνςφϯτ

    ։ൃऀϑϨϯυϦʔ ϦΞϧλΠϜɺશจݕࡧ ΞάϦήʔγϣϯ
  28. Elasticsearchͱ͸ʁ

  29. !29 ϑϦʔϫʔυݕࡧ

  30. !30 ߜΓࠐΈ

  31. !31 ϋΠϥΠτ

  32. !32 ιʔτ

  33. !33 ϖʔδϯά

  34. !34 ूܭ

  35. !35 αδΣετ

  36. !36 Elasticsearch in 10 seconds • εΩʔϚϑϦʔɺ෼ࢄυΩϡϝϯτετΞɺREST & JSON •

    Φʔϓϯιʔε: Apache License 2.0 • ઃఆͳ͠Ͱ؆୯ʹࢼ͢͜ͱ͕Մೳ • JavaͰ࣮૷ɻ֦ு΋༰қ
  37. ؆୯ͳCRUD

  38. σʔλొ࿥ 38 curl -XPUT localhost:9200/books/doc/1 -d ' { "title" :

    "Elasticsearch - The definitive guide", "authors" : "Clinton Gormley", "started" : "2013-02-04", "pages" : 230 }'
  39. σʔλߋ৽ 39 curl -XPUT localhost:9200/books/doc/1 -d ' { "title" :

    "Elasticsearch - The definitive guide", "authors" : [ "Clinton Gormley", "Zachary Tong" ], "started" : "2013-02-04", "pages" : 230 }'
  40. σʔλ࡟আ !40 curl -X DELETE localhost:9200/books/doc/1 σʔλͷऔಘ curl —X GET

    localhost:9200/books/doc/1 curl —X GET localhost:9200/books/doc/1/_source
  41. ݕࡧ - Query DSL !41 curl -XGET ‘localhost:9200/books/doc/_search' -d '{

    "query": { "bool": { "must": [ { "match": { "title": "Search" }}, { "match": { "content": "Elasticsearch" }} ], "filter": [ { "term": { "status": "published" }}, { "range": { "publish_date": { "gte": "2015-01-01" }}} ] } } }'
  42. ෼ࢄߏ੒ɺ
 εέʔϧ

  43. Basic terms • ΠϯσοΫε ‒ σʔλͷ࿦ཧతͳू߹ɻ
 RDBͷσʔλϕʔεͷΑ͏ͳ΋ͷLogical • ϨϓϦέʔγϣϯ •

    ಡΈࠐΈͷεέʔϥϏϦςΟ޲্ • SPOFͷղফ • γϟʔσΟϯά • ෳ਺Ϛγϯ΁σʔλΛ෼ׂ
 ॻ͖ࠐΈͷεέʔϥϏϦςΟ޲্
 σʔλϑϩʔ੍ޚ !43
  44. γϟʔυͱϨϓϦΧ !44 node 1 orders products 1 4 1 2

    2 3 curl -X PUT localhost:9200/orders -d '{ "settings.index.number_of_shards" : 4 "settings.index.number_of_replicas" : 1 }' curl -X PUT localhost:9200/products -d '{ "settings.index.number_of_shards" : 2 "settings.index.number_of_replicas" : 0 }'
  45. γϟʔυͱϨϓϦΧ !45 node 1 orders products 1 4 1 node

    2 orders products 2 2 3 4 1 2 3
  46. ࣗಈతͳ෼ࢄ !46 node 1 orders products 2 1 4 1

    node 2 orders products 2 2 node 3 orders products 3 4 1 3
  47. ͦͷଞͷػೳ

  48. elasticsearch ͞·͟·ͳܗࣜͷσʔλͰ GeoݕࡧՄೳ
 
 Ң౓ܦ౓ɺGeoHashɺ GeoShape… GEO

  49. Ecosystem • Plugins ‒ ϓϥάΠϯʹΑΔػೳͷ௥Ճ • ΫϥΠΞϯτϥΠϒϥϦ • Java, Ruby,

    python, php, perl, javascript, .NET • Scala, clojure, go !49
  50. Elasticsearch - The Definitive guide
 
 http://www.elastic.co/guide/en/ elasticsearch/guide/current/index.html 50 ৄ͘͠஌Γ͍ͨํ͸

  51. !51

  52. 52 Kibana Window into the Elastic Stack ՄࢹԽͱ෼ੳ ஍ཧۭؒ ΧελϚΠζͱ

    Ϩϙʔτͷڞ༗ άϥϑ୳ࡧ Elastic Stack΁ͷ ηΩϡΞͳΞΫηεͱ؅ཧ ΧελϜAppsͷ࡞੒
  53. !53 Kibana 6

  54. !54 σϞ σʔλ౤ೖ͔ΒՄࢹԽ·Ͱ

  55. ຊ֨తʹղੳΛߦ͏ʹ͸ʁ

  56. !56

  57. 57 Logstash σʔλՃ޻ύΠϓϥΠϯ શͯͷܗࣜɺαΠζͱσʔλιʔ εͷ౤ೖ ύʔεͱಈతͳ σʔλม׵ ͋ΒΏΔग़ྗʹ σʔλసૹ ҆શͰ҉߸Խ͞Εͨ


    σʔλೖྗ ಠࣗͷύΠϓϥΠϯॲཧ ͷ࡞੒ 200Ҏ্ͷϓϥάΠϯ
  58. Logstash architecture !58 Input Output Filter ? ? collect and

    split alter and enrich store and visualize
  59. ઃఆ 59 input { … } filter { … }

    output { … }
  60. 1ߦ1σʔλ 189.120.xx.xx - - [02/Dec/2014:12:18:29 +0900] "GET /manager/html HTTP/ 1.1"

    404 274 "-" "Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0" 60
  61. ઃఆɿfilter 61 filter { grok { match => { "message"

    => "%{COMBINEDAPACHELOG}" } break_on_match => false } date { match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"] locale => en } geoip { source => ["clientip"] } useragent { source => "agent" target => "useragent" } }
  62. ύʔε !62 189.120.xx.xx - - [02/Dec/2014:12:18:29 +0900] "GET /manager/html HTTP/1.1"

    404 274 "-" "Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0" {… "@timestamp": "2015-04-10T09:07:49.325Z", "clientip": "189.120.xx.xx", "ident": "-", "auth": "-", "timestamp": "02/Dec/2014:12:18:29 +0900", "verb": "GET", "request": "/manager/html", … "agent": "\"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/
  63. ઃఆɿfilter !63 filter { grok { match => { "message"

    => "%{COMBINEDAPACHELOG}" } break_on_match => false } date { match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"] locale => en } geoip { source => ["clientip"] } useragent { source => "agent" target => "useragent" } }
  64. ೔෇ͷύʔε 64 {… "@timestamp": "2015-04-10T09:07:49.325Z", … "timestamp": "02/Dec/2014:12:18:29 +0900", …

    } {… "@timestamp": "2014-12-02T03:18:29.000Z", … "timestamp": "02/Dec/2014:12:18:29 +0900", … }
  65. ઃఆɿfilter !65 filter { grok { match => { "message"

    => "%{COMBINEDAPACHELOG}" } break_on_match => false } date { match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"] locale => en } geoip { source => ["clientip"] } useragent { source => "agent" target => "useragent" } }
  66. IP͔ΒҢ౓ܦ౓ͳͲ෇༩ 66 "clientip": "189.120.xx.xx", "clientip": "189.120.xx.xx", "geoip": { "ip": “189.120.xxx.xxx”,

    … "country_name": "Brazil", "continent_code": "SA", "region_name": "27", "city_name": "São Paulo", "latitude":
  67. ઃఆɿfilter !67 filter { grok { match => { "message"

    => "%{COMBINEDAPACHELOG}" } break_on_match => false } date { match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"] locale => en } geoip { source => ["clientip"] } useragent { source => "agent" target => "useragent" } }
  68. ϢʔβΤʔδΣϯτͷύʔε 68 "agent": "\"Mozilla/5.0 (Windows NT 5.1; rv: 5.0) Gecko/20100101

    Firefox/5.0\"" "agent": "\"Mozilla/5.0 (Windows NT 5.1; rv: 5.0) Gecko/20100101 Firefox/5.0\"" "useragent": { "name": "Firefox", "os": "Windows XP", "os_name": "Windows XP", "device": "Other", "major": "5", "minor": "0"
  69. ͦͷ΄͔ʹ͸ʁ

  70. !70 elasticsearch-hadoop - •  D E H •  PD ecd

    ER •  g D •  CH •  Ca M DMS D FERC
  71. !71

  72. !72 X-Pack ؆୯ʹΠϯετʔϧ Elastic StackΛ֦ு αϒεΫϦϓγϣϯʹؚ·ΕΔ Security Alerting Monitoring Reporting

    Graph Machine Learning
  73. !73

  74. !74 ࢀߟจݙ • Elasticsearch - The Definitive guide ‒ http://www.elastic.co/guide/en/elasticsearch/guide/current/

    index.html • ॻ੶ʢ೔ຊޠʣ ‒ σʔλ෼ੳج൫ߏஙೖ໳ ‒ Elasticsearch࣮ફΨΠυ
  75. !75 ࢀߟαΠτ • Ϣʔεέʔε • https://www.elastic.co/use-cases • DiscussʢWebϑΥʔϥϜʣ • https://discuss.elastic.co

    • Elastic{ON}ͷϏσΦͱࢿྉ • https://www.elastic.co/elasticon/videos • αϙʔτϝχϡʔ • https://www.elastic.co/subscriptions
  76. Thanks for listening! Q & A We’re hiring! https://www.elastic.co/about/careers/ We’re

    helping! https://www.elastic.co/subscriptions http://training.elastic.co