Elastic Stackを利用して データから様々な気づきを見つける

#BigDataTokyo BigData Analytics Tokyoでの発表スライドです。

Transcript

1. ‹#› 2017/02/07 Evangelist at Elastic Jun Ohtani @johtani Elastic StackΛར༻ͯ͠ σʔλ͔Β༷ʑͳؾ͖ͮΛݟ͚ͭΔ
2. ‹#›
3. ΞδΣϯμ • ؾ͖ͮΛݟ͚ͭΔͱ͸ʁ • Ϣʔεέʔεͷ঺հ • Elastic stack঺հ • BeatsɺLogstashɺElasticsearchɺKibanaɺX-Pack • σϞ 3
4. about • Me, Jun Ohtani / Technical Advocate ‒ lucene-gosenίϛολʔ ‒ ElasticSearch Server೔ຊޠ൛ͷ຋༁ ‒ http://blog.johtani.info
 • Elasticsearch, founded in 2012 ‒ Products: Elasticsearch, Logstash, Kibana, Beats 
 X-Pack, Elastic Cloud
 Professional services: Support & development subscriptions ‒ Trainings & Consulting 4
5. ༷ʑͳϢʔεέʔε 5 ؾ͖ͮΛݟ͚ͭΔ ͱ͸ʁ
6. Search and analytics, it all started here More than 60% of our customers have a search or analytics use case
9. Logs Logs Logs, 
 many devices,
 many systems More than 40% of our
 customers use our products
 for operational log analysis
10. We collect more than 1.2 TB logs every day from our infrastructure, web servers, and applications. 10
11. 11 We handle more than 3 Billion daily events while meeting our all of our data security requirements.
12. Sniff sniff sniff,
 find the bad actors
 in your data 200% YoY growth in security use cases with our products
13. We analyze piles of data: 13B AMP queries/day 600B emails/day 16B web requests/day 13
14. 14 We mine and analyze 4 billion events every day to detect security hacks and threats. 1
15. The Elastic Stack: 
 A foundation to solve many use cases 75% of our customers use our products for more than one use case SEARCH SECURIT CUSTOM APPS METRICS OPERATIONAL
 ANALYTICS LOG ANALYSIS
16. Operational analytics Flight telemetry analysis Anomaly resolution Internal search engine 16
17. 17 Enterprise search Intranet search Real-time log analytics Legal contract repository Trade tracking application HR recruiting application
18. 18 ElasticελοΫ
19. ElasticελοΫʢOpen Sourceʣ 19 Kibana    
    Elasticsearch    
    
    Logstash Beats
20. ElasticελοΫ 20 Elastic Cloud  
    
    
     X-Pack Kibana    
    Elasticsearch !  "
    
    Logstash Beats +
21. Ingest
22. 22 Logstash
23. Logstash in 10 seconds • ϩάɾσʔλͷऩूɾ؅ཧ • ऩूɺύʔεɾՃ޻ɺૹग़ • ΦʔϓϯιʔεɿApache License 2.0 • Ruby app (JRuby) 23
24. Logstash architecture 24 Input Output Filter ? ? collect and split alter and enrich store and visualize
25. ઃఆ 25 input { … } filter { … } output { … }
26. ઃఆɿinput 26 input { file { path => “/Users/johtani/sample/*_log" start_position => "beginning" } }
27. 1ߦ1σʔλ 189.120.xx.xx - - [02/Dec/2014:12:18:29 +0900] "GET /manager/html HTTP/ 1.1" 404 274 "-" "Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0" 27
28. ઃఆɿfilter 28 filter { grok { match => { "message" => "%{COMBINEDAPACHELOG}" } break_on_match => false } date { match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"] locale => en } geoip { source => ["clientip"] } useragent { source => "agent" target => "useragent" } }
29. ύʔε 29 189.120.xx.xx - - [02/Dec/2014:12:18:29 +0900] "GET /manager/html HTTP/1.1" 404 274 "-" "Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0" {… "@timestamp": "2015-04-10T09:07:49.325Z", "clientip": "189.120.xx.xx", "ident": "-", "auth": "-", "timestamp": "02/Dec/2014:12:18:29 +0900", "verb": "GET", "request": "/manager/html", … "agent": "\"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/
30. ઃఆɿfilter 30 filter { grok { match => { "message" => "%{COMBINEDAPACHELOG}" } break_on_match => false } date { match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"] locale => en } geoip { source => ["clientip"] } useragent { source => "agent" target => "useragent" } }
31. IP͔ΒҢ౓ܦ౓ͳͲ෇༩ 31 "clientip": "189.120.xx.xx", "clientip": "189.120.xx.xx", "geoip": { "ip": “189.120.xxx.xxx”, … "country_name": "Brazil", "continent_code": "SA", "region_name": "27", "city_name": "São Paulo", "latitude":
32. ઃఆɿoutput 32 output { elasticsearch { hosts => ["localhost"] index => “demo_access_log-%{+YYYY.MM.dd}” } }
33. ܰྔσʔλγούʔ 33 Beats
34. To tail a File filebeat
35. To tail a File filebeat
36. Capture the Packet Packetbeat
37. Capture the Packet Packetbeat
38. Welcome to 1998 winlogbeat
39. Now winlogbeat
40. Store, Search & Analytics
41. 41 Elasticsearch
42. ݕࡧͱͯ͠ͷ
 Elasticsearch
43. Elasticsearchͱ͸ʁ
44. ϑϦʔϫʔυݕࡧ 44
45. ߜΓࠐΈ 45
46. ϋΠϥΠτ 46
47. ιʔτ 47
48. ϖʔδϯά 48
49. ूܭ 49
50. αδΣετ 50
51. Elasticsearch in 10 seconds • εΩʔϚϑϦʔɺ෼ࢄυΩϡϝϯτετΞɺREST & JSON • Φʔϓϯιʔε: Apache License 2.0 • ઃఆͳ͠Ͱ؆୯ʹࢼ͢͜ͱ͕Մೳ • JavaͰ࣮૷ɻ֦ு΋༰қ 51
52. ղੳͱͯ͠ͷ
 Elasticsearch
53. aggregation
54. Aggregationͱ͸ • 1.0͔Βಋೖ • FacetΑΓ΋ڧྗͳूܭͳͲ͕Մೳ • ֊૚తͳूܭɺάϧʔϓԽ
 ಈతͳूܭɺάϧʔϓԽ • େ͖͘2छྨ • BucketɹυΩϡϝϯτΛ஋͝ͱʹ݁ՌΛάϧʔϐϯά • Metricɹ υΩϡϝϯτͷ࣋ͭ஋Λूܭ 54
55. ྫɿݴޠ͓Αͼ஍Ҭͷूܭ 55 curl -XGET twitter-2014.08.22/_search -d ' { "aggs": { "lang": { "terms": {"field": "lang" }, "aggs": { "place": { "terms": { "field": “place.full_name", "size": 10 } } } } } }
56. ྫɿݴޠ͓Αͼ஍Ҭͷूܭ 56 "aggregations": { "lang": { "buckets": [{…}, { "key": "ja", "doc_count": 980145, "place": { "buckets": [ { "key": "ژ౎ࢢ෬ݟ۠, ژ౎", "doc_count":252 }, { "key": "ઍ୅ా۠, ౦ژ", "doc_count": 39 },…
57. elasticsearch-hadoop 57 - •  D E H •  PD ecd ER •  g D •  CH •  Ca M DMS D FERC
58. The Window into the Elastic Stack
59. 59 KibanaͰՄࢹԽ
60. Kibana 5 • ElasticsearchͷσʔλΛՄࢹԽ • Node.js server & JavaScript • Apache License 2.0 • Elastic Stackͷ૭ͷ໾ׂ • ༷ʑͳGUIΛPluginͱ͍ͯެ։ • MarvelɺSenseɺTimelionͳͲ 60
61. Kibana 5 61
63. X-Pack 5.0: Extending the Elastic Stack
64. Security
65. X-Pack : Securityͷಛ௃ • User Authentication ‒ LDAP/Active Directory/ϑΝΠϧϕʔε • Authorization ‒ ϩʔϧϕʔεͷΞΫηείϯτϩʔϧ ‒ ΠϯσοΫε͝ͱɺΞΫγϣϯ͝ͱͷઃఆ͕Մೳ ‒ υΩϡϝϯτɾϑΟʔϧυ͝ͱͷઃఆ΋Մೳʹ • ηΩϡΞͳ௨৴ ‒ ElasticsearchϊʔυؒͷSSL/TLSɺIPϑΟϧλϦϯά • ؂ࠪϩά 65
66. Alerting
67. X-Pack : Alertingͷಛ௃ • ΫΤϦʹΑΔWatch ‒ ElasticsearchͷΫΤϦΛར༻ͯ͠σʔλͷ؂ࢹ • ৚݅ͷઃఆ ‒ ΞΫγϣϯΛ࣮ߦ͢Δ͔Ͳ͏͔ͷઃఆ • εέδϡʔϧ ‒ ΫΤϦΛ࣮ߦ͠ɺ৚݅ΛνΣοΫ͢Δස౓ͷࢦఆ • ΞΫγϣϯͷఆٛ ‒ ϝʔϧͷૹ৴ɺଞγεςϜ΁ͷσʔλૹ৴ͳͲͷಈ࡞Λઃఆ • ཤྺͷอଘ 67
68. Graph
69. Graphͷಛ௃ • σʔλؒͷͭͳ͕ΓΛ୳ࡧ͢ΔϓϥάΠϯ • KibanaϓϥάΠϯʹΑΓGUIΛར༻ͯ͠୳ࡧՄೳ 69
70. Prelert
71. σʔλ͔Β༗ҙٛͳ৘ใΛݟ͚ͭΔํ๏
    Search Aggregations Visualization Machine Learning
72. 1SFMFSUͷςΫϊϩδʔ
    σʔλʹજΉߦಈϞσϧΛ
 ࣗಈతʹڭࢣͳֶ͠श ݱࡏͷߦಈ͕༧ଌϞσϧͱ
 ݦஶʹҟͳΔ৔߹ʹ௨஌
73. 73 σϞ Demo
74. ࢀߟαΠτ • Ϣʔεέʔε • https://www.elastic.co/use-cases • DiscussʢWebϑΥʔϥϜʣ • https://discuss.elastic.co • Elastic{ON}ͷϏσΦͱࢿྉ • https://www.elastic.co/elasticon/videos • αϙʔτϝχϡʔ • https://www.elastic.co/subscriptions 74
75. 75 March 7-9, 2017 • Pier 48 • San Francisco, CA • 2,500 attendees 3rd Annual Elastic User Conference Topics • Latest Roadmap • Ask Me Anything Booth • 70+ Sessions • 76 Demo Hours
76. Ξϯέʔτ΁ͷճ౴Λ͓ئ͍͠·͢ bit.ly/bigdata-tokyo-elastic
77. Thanks for listening! Q & A We’re hiring! https://www.elastic.co/about/careers/ We’re helping! https://www.elastic.co/subscriptions http://training.elastic.co