Pro Yearly is on sale from $80 to $50! »

Elastic Stackを利用して データから様々な気づきを見つける

657aeeff3fc467567dacebf8a1ea0b23?s=47 Jun Ohtani
February 07, 2017

Elastic Stackを利用して データから様々な気づきを見つける

#BigDataTokyo BigData Analytics Tokyoでの発表スライドです。

657aeeff3fc467567dacebf8a1ea0b23?s=128

Jun Ohtani

February 07, 2017
Tweet

Transcript

  1. ‹#› 2017/02/07 Evangelist at Elastic Jun Ohtani @johtani Elastic StackΛར༻ͯ͠

    σʔλ͔Β༷ʑͳؾ͖ͮΛݟ͚ͭΔ
  2. ‹#›

  3. ΞδΣϯμ • ؾ͖ͮΛݟ͚ͭΔͱ͸ʁ • Ϣʔεέʔεͷ঺հ • Elastic stack঺հ • BeatsɺLogstashɺElasticsearchɺKibanaɺX-Pack

    • σϞ 3
  4. about • Me, Jun Ohtani / Technical Advocate ‒ lucene-gosenίϛολʔ

    ‒ ElasticSearch Server೔ຊޠ൛ͷ຋༁ ‒ http://blog.johtani.info
 • Elasticsearch, founded in 2012 ‒ Products: Elasticsearch, Logstash, Kibana, Beats 
 X-Pack, Elastic Cloud
 Professional services: Support & development subscriptions ‒ Trainings & Consulting 4
  5. ༷ʑͳϢʔεέʔε 5 ؾ͖ͮΛݟ͚ͭΔ ͱ͸ʁ

  6. Search and analytics, it all started here More than 60%

    of our customers have a search or analytics use case
  7. 7

  8. 8

  9. Logs Logs Logs, 
 many devices,
 many systems More than

    40% of our
 customers use our products
 for operational log analysis
  10. We collect more than 1.2 TB logs every day from

    our infrastructure, web servers, and applications. 10
  11. 11 We handle more than 3 Billion daily events while

    meeting our all of our data security requirements.
  12. Sniff sniff sniff,
 find the bad actors
 in your data

    200% YoY growth in security use cases with our products
  13. We analyze piles of data: 13B AMP queries/day 600B emails/day

    16B web requests/day 13
  14. 14 We mine and analyze 4 billion events every day

    to detect security hacks and threats. 1
  15. The Elastic Stack: 
 A foundation to solve many use

    cases 75% of our customers use our products for more than one use case SEARCH SECURIT CUSTOM APPS METRICS OPERATIONAL
 ANALYTICS LOG ANALYSIS
  16. Operational analytics Flight telemetry analysis Anomaly resolution Internal search engine

    16
  17. 17 Enterprise search Intranet search Real-time log analytics Legal contract

    repository Trade tracking application HR recruiting application
  18. 18 ElasticελοΫ

  19. ElasticελοΫʢOpen Sourceʣ 19 Kibana      Elasticsearch

         Logstash Beats
  20. ElasticελοΫ 20 Elastic Cloud      

    X-Pack Kibana     Elasticsearch !  "  Logstash Beats +
  21. Ingest

  22. 22 Logstash

  23. Logstash in 10 seconds • ϩάɾσʔλͷऩूɾ؅ཧ • ऩूɺύʔεɾՃ޻ɺૹग़ • ΦʔϓϯιʔεɿApache

    License 2.0 • Ruby app (JRuby) 23
  24. Logstash architecture 24 Input Output Filter ? ? collect and

    split alter and enrich store and visualize
  25. ઃఆ 25 input { … } filter { … }

    output { … }
  26. ઃఆɿinput 26 input { file { path => “/Users/johtani/sample/*_log" start_position

    => "beginning" } }
  27. 1ߦ1σʔλ 189.120.xx.xx - - [02/Dec/2014:12:18:29 +0900] "GET /manager/html HTTP/ 1.1"

    404 274 "-" "Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0" 27
  28. ઃఆɿfilter 28 filter { grok { match => { "message"

    => "%{COMBINEDAPACHELOG}" } break_on_match => false } date { match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"] locale => en } geoip { source => ["clientip"] } useragent { source => "agent" target => "useragent" } }
  29. ύʔε 29 189.120.xx.xx - - [02/Dec/2014:12:18:29 +0900] "GET /manager/html HTTP/1.1"

    404 274 "-" "Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0" {… "@timestamp": "2015-04-10T09:07:49.325Z", "clientip": "189.120.xx.xx", "ident": "-", "auth": "-", "timestamp": "02/Dec/2014:12:18:29 +0900", "verb": "GET", "request": "/manager/html", … "agent": "\"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/
  30. ઃఆɿfilter 30 filter { grok { match => { "message"

    => "%{COMBINEDAPACHELOG}" } break_on_match => false } date { match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"] locale => en } geoip { source => ["clientip"] } useragent { source => "agent" target => "useragent" } }
  31. IP͔ΒҢ౓ܦ౓ͳͲ෇༩ 31 "clientip": "189.120.xx.xx", "clientip": "189.120.xx.xx", "geoip": { "ip": “189.120.xxx.xxx”,

    … "country_name": "Brazil", "continent_code": "SA", "region_name": "27", "city_name": "São Paulo", "latitude":
  32. ઃఆɿoutput 32 output { elasticsearch { hosts => ["localhost"] index

    => “demo_access_log-%{+YYYY.MM.dd}” } }
  33. ܰྔσʔλγούʔ 33 Beats

  34. To tail a File filebeat

  35. To tail a File filebeat

  36. Capture the Packet Packetbeat

  37. Capture the Packet Packetbeat

  38. Welcome to 1998 winlogbeat

  39. Now winlogbeat

  40. Store, Search & Analytics

  41. 41 Elasticsearch

  42. ݕࡧͱͯ͠ͷ
 Elasticsearch

  43. Elasticsearchͱ͸ʁ

  44. ϑϦʔϫʔυݕࡧ 44

  45. ߜΓࠐΈ 45

  46. ϋΠϥΠτ 46

  47. ιʔτ 47

  48. ϖʔδϯά 48

  49. ूܭ 49

  50. αδΣετ 50

  51. Elasticsearch in 10 seconds • εΩʔϚϑϦʔɺ෼ࢄυΩϡϝϯτετΞɺREST & JSON • Φʔϓϯιʔε:

    Apache License 2.0 • ઃఆͳ͠Ͱ؆୯ʹࢼ͢͜ͱ͕Մೳ • JavaͰ࣮૷ɻ֦ு΋༰қ 51
  52. ղੳͱͯ͠ͷ
 Elasticsearch

  53. aggregation

  54. Aggregationͱ͸ • 1.0͔Βಋೖ • FacetΑΓ΋ڧྗͳूܭͳͲ͕Մೳ • ֊૚తͳूܭɺάϧʔϓԽ
 ಈతͳूܭɺάϧʔϓԽ • େ͖͘2छྨ

    • BucketɹυΩϡϝϯτΛ஋͝ͱʹ݁ՌΛάϧʔϐϯά • Metricɹ υΩϡϝϯτͷ࣋ͭ஋Λूܭ 54
  55. ྫɿݴޠ͓Αͼ஍Ҭͷूܭ 55 curl -XGET twitter-2014.08.22/_search -d ' { "aggs": {

    "lang": { "terms": {"field": "lang" }, "aggs": { "place": { "terms": { "field": “place.full_name", "size": 10 } } } } } }
  56. ྫɿݴޠ͓Αͼ஍Ҭͷूܭ 56 "aggregations": { "lang": { "buckets": [{…}, { "key":

    "ja", "doc_count": 980145, "place": { "buckets": [ { "key": "ژ౎ࢢ෬ݟ۠, ژ౎", "doc_count":252 }, { "key": "ઍ୅ా۠, ౦ژ", "doc_count": 39 },…
  57. elasticsearch-hadoop 57 - •  D E H •  PD ecd

    ER •  g D •  CH •  Ca M DMS D FERC
  58. The Window into the Elastic Stack

  59. 59 KibanaͰՄࢹԽ

  60. Kibana 5 • ElasticsearchͷσʔλΛՄࢹԽ • Node.js server & JavaScript •

    Apache License 2.0 • Elastic Stackͷ૭ͷ໾ׂ • ༷ʑͳGUIΛPluginͱ͍ͯެ։ • MarvelɺSenseɺTimelionͳͲ 60
  61. Kibana 5 61

  62. None
  63. X-Pack 5.0: Extending the Elastic Stack

  64. Security

  65. X-Pack : Securityͷಛ௃ • User Authentication ‒ LDAP/Active Directory/ϑΝΠϧϕʔε •

    Authorization ‒ ϩʔϧϕʔεͷΞΫηείϯτϩʔϧ ‒ ΠϯσοΫε͝ͱɺΞΫγϣϯ͝ͱͷઃఆ͕Մೳ ‒ υΩϡϝϯτɾϑΟʔϧυ͝ͱͷઃఆ΋Մೳʹ • ηΩϡΞͳ௨৴ ‒ ElasticsearchϊʔυؒͷSSL/TLSɺIPϑΟϧλϦϯά • ؂ࠪϩά 65
  66. Alerting

  67. X-Pack : Alertingͷಛ௃ • ΫΤϦʹΑΔWatch ‒ ElasticsearchͷΫΤϦΛར༻ͯ͠σʔλͷ؂ࢹ • ৚݅ͷઃఆ ‒

    ΞΫγϣϯΛ࣮ߦ͢Δ͔Ͳ͏͔ͷઃఆ • εέδϡʔϧ ‒ ΫΤϦΛ࣮ߦ͠ɺ৚݅ΛνΣοΫ͢Δස౓ͷࢦఆ • ΞΫγϣϯͷఆٛ ‒ ϝʔϧͷૹ৴ɺଞγεςϜ΁ͷσʔλૹ৴ͳͲͷಈ࡞Λઃఆ • ཤྺͷอଘ 67
  68. Graph

  69. Graphͷಛ௃ • σʔλؒͷͭͳ͕ΓΛ୳ࡧ͢ΔϓϥάΠϯ • KibanaϓϥάΠϯʹΑΓGUIΛར༻ͯ͠୳ࡧՄೳ 69

  70. Prelert

  71. σʔλ͔Β༗ҙٛͳ৘ใΛݟ͚ͭΔํ๏ Search Aggregations Visualization Machine Learning

  72. 1SFMFSUͷςΫϊϩδʔ σʔλʹજΉߦಈϞσϧΛ
 ࣗಈతʹڭࢣͳֶ͠श ݱࡏͷߦಈ͕༧ଌϞσϧͱ
 ݦஶʹҟͳΔ৔߹ʹ௨஌

  73. 73 σϞ Demo

  74. ࢀߟαΠτ • Ϣʔεέʔε • https://www.elastic.co/use-cases • DiscussʢWebϑΥʔϥϜʣ • https://discuss.elastic.co •

    Elastic{ON}ͷϏσΦͱࢿྉ • https://www.elastic.co/elasticon/videos • αϙʔτϝχϡʔ • https://www.elastic.co/subscriptions 74
  75. 75 March 7-9, 2017 • Pier 48 • San Francisco,

    CA • 2,500 attendees 3rd Annual Elastic User Conference Topics • Latest Roadmap • Ask Me Anything Booth • 70+ Sessions • 76 Demo Hours
  76. Ξϯέʔτ΁ͷճ౴Λ͓ئ͍͠·͢ bit.ly/bigdata-tokyo-elastic

  77. Thanks for listening! Q & A We’re hiring! https://www.elastic.co/about/careers/ We’re

    helping! https://www.elastic.co/subscriptions http://training.elastic.co