Elastic Stackを利用して データから様々な気づきを見つける

Transcript

1. ‹#› 2017/02/07 Evangelist at Elastic Jun Ohtani @johtani Elastic StackΛར༻ͯ͠

   σʔλ͔Β༷ʑͳؾ͖ͮΛݟ͚ͭΔ

2. ‹#›

3. ΞδΣϯμ • ؾ͖ͮΛݟ͚ͭΔͱ͸ʁ • Ϣʔεέʔεͷ঺հ • Elastic stack঺հ • BeatsɺLogstashɺElasticsearchɺKibanaɺX-Pack

   • σϞ 3

4. about • Me, Jun Ohtani / Technical Advocate ‒ lucene-gosenίϛολʔ

   ‒ ElasticSearch Server೔ຊޠ൛ͷ຋༁ ‒ http://blog.johtani.info
 • Elasticsearch, founded in 2012 ‒ Products: Elasticsearch, Logstash, Kibana, Beats 
 X-Pack, Elastic Cloud
 Professional services: Support & development subscriptions ‒ Trainings & Consulting 4

5. ༷ʑͳϢʔεέʔε 5 ؾ͖ͮΛݟ͚ͭΔ ͱ͸ʁ

6. Search and analytics, it all started here More than 60%

   of our customers have a search or analytics use case

7. 7

8. 8

9. Logs Logs Logs, 
 many devices,
 many systems More than

   40% of our
 customers use our products
 for operational log analysis

10. We collect more than 1.2 TB logs every day from

    our infrastructure, web servers, and applications. 10

11. 11 We handle more than 3 Billion daily events while

    meeting our all of our data security requirements.

12. Sniff sniff sniff,
 find the bad actors
 in your data

    200% YoY growth in security use cases with our products

13. We analyze piles of data: 13B AMP queries/day 600B emails/day

    16B web requests/day 13

14. 14 We mine and analyze 4 billion events every day

    to detect security hacks and threats. 1

15. The Elastic Stack: 
 A foundation to solve many use

    cases 75% of our customers use our products for more than one use case SEARCH SECURIT CUSTOM APPS METRICS OPERATIONAL
 ANALYTICS LOG ANALYSIS

16. Operational analytics Flight telemetry analysis Anomaly resolution Internal search engine

    16

17. 17 Enterprise search Intranet search Real-time log analytics Legal contract

    repository Trade tracking application HR recruiting application

18. 18 ElasticελοΫ

19. ElasticελοΫʢOpen Sourceʣ 19 Kibana    
    Elasticsearch

       
    
    Logstash Beats

20. ElasticελοΫ 20 Elastic Cloud  
    
    
    

    X-Pack Kibana    
    Elasticsearch !  "
    
    Logstash Beats +

21. Ingest

22. 22 Logstash

23. Logstash in 10 seconds • ϩάɾσʔλͷऩूɾ؅ཧ • ऩूɺύʔεɾՃ޻ɺૹग़ • ΦʔϓϯιʔεɿApache

    License 2.0 • Ruby app (JRuby) 23

24. Logstash architecture 24 Input Output Filter ? ? collect and

    split alter and enrich store and visualize

25. ઃఆ 25 input { … } filter { … }

    output { … }

26. ઃఆɿinput 26 input { file { path => “/Users/johtani/sample/*_log" start_position

    => "beginning" } }

27. 1ߦ1σʔλ 189.120.xx.xx - - [02/Dec/2014:12:18:29 +0900] "GET /manager/html HTTP/ 1.1"

    404 274 "-" "Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0" 27

28. ઃఆɿfilter 28 filter { grok { match => { "message"

    => "%{COMBINEDAPACHELOG}" } break_on_match => false } date { match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"] locale => en } geoip { source => ["clientip"] } useragent { source => "agent" target => "useragent" } }

29. ύʔε 29 189.120.xx.xx - - [02/Dec/2014:12:18:29 +0900] "GET /manager/html HTTP/1.1"

    404 274 "-" "Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0" {… "@timestamp": "2015-04-10T09:07:49.325Z", "clientip": "189.120.xx.xx", "ident": "-", "auth": "-", "timestamp": "02/Dec/2014:12:18:29 +0900", "verb": "GET", "request": "/manager/html", … "agent": "\"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/

30. ઃఆɿfilter 30 filter { grok { match => { "message"

    => "%{COMBINEDAPACHELOG}" } break_on_match => false } date { match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"] locale => en } geoip { source => ["clientip"] } useragent { source => "agent" target => "useragent" } }

31. IP͔ΒҢ౓ܦ౓ͳͲ෇༩ 31 "clientip": "189.120.xx.xx", "clientip": "189.120.xx.xx", "geoip": { "ip": “189.120.xxx.xxx”,

    … "country_name": "Brazil", "continent_code": "SA", "region_name": "27", "city_name": "São Paulo", "latitude":

32. ઃఆɿoutput 32 output { elasticsearch { hosts => ["localhost"] index

    => “demo_access_log-%{+YYYY.MM.dd}” } }

33. ܰྔσʔλγούʔ 33 Beats

34. To tail a File filebeat

35. To tail a File filebeat

36. Capture the Packet Packetbeat

37. Capture the Packet Packetbeat

38. Welcome to 1998 winlogbeat

39. Now winlogbeat

40. Store, Search & Analytics

41. 41 Elasticsearch

42. ݕࡧͱͯ͠ͷ
 Elasticsearch

43. Elasticsearchͱ͸ʁ

44. ϑϦʔϫʔυݕࡧ 44

45. ߜΓࠐΈ 45

46. ϋΠϥΠτ 46

47. ιʔτ 47

48. ϖʔδϯά 48

49. ूܭ 49

50. αδΣετ 50

51. Elasticsearch in 10 seconds • εΩʔϚϑϦʔɺ෼ࢄυΩϡϝϯτετΞɺREST & JSON • Φʔϓϯιʔε:

    Apache License 2.0 • ઃఆͳ͠Ͱ؆୯ʹࢼ͢͜ͱ͕Մೳ • JavaͰ࣮૷ɻ֦ு΋༰қ 51

52. ղੳͱͯ͠ͷ
 Elasticsearch

53. aggregation

54. Aggregationͱ͸ • 1.0͔Βಋೖ • FacetΑΓ΋ڧྗͳूܭͳͲ͕Մೳ • ֊૚తͳूܭɺάϧʔϓԽ
 ಈతͳूܭɺάϧʔϓԽ • େ͖͘2छྨ

    • BucketɹυΩϡϝϯτΛ஋͝ͱʹ݁ՌΛάϧʔϐϯά • Metricɹ υΩϡϝϯτͷ࣋ͭ஋Λूܭ 54

55. ྫɿݴޠ͓Αͼ஍Ҭͷूܭ 55 curl -XGET twitter-2014.08.22/_search -d ' { "aggs": {

    "lang": { "terms": {"field": "lang" }, "aggs": { "place": { "terms": { "field": “place.full_name", "size": 10 } } } } } }

56. ྫɿݴޠ͓Αͼ஍Ҭͷूܭ 56 "aggregations": { "lang": { "buckets": [{…}, { "key":

    "ja", "doc_count": 980145, "place": { "buckets": [ { "key": "ژ౎ࢢ෬ݟ۠, ژ౎", "doc_count":252 }, { "key": "ઍ୅ా۠, ౦ژ", "doc_count": 39 },…

57. elasticsearch-hadoop 57 - •  D E H •  PD ecd

    ER •  g D •  CH •  Ca M DMS D FERC

58. The Window into the Elastic Stack

59. 59 KibanaͰՄࢹԽ

60. Kibana 5 • ElasticsearchͷσʔλΛՄࢹԽ • Node.js server & JavaScript •

    Apache License 2.0 • Elastic Stackͷ૭ͷ໾ׂ • ༷ʑͳGUIΛPluginͱ͍ͯެ։ • MarvelɺSenseɺTimelionͳͲ 60

61. Kibana 5 61

62. None

63. X-Pack 5.0: Extending the Elastic Stack

64. Security

65. X-Pack : Securityͷಛ௃ • User Authentication ‒ LDAP/Active Directory/ϑΝΠϧϕʔε •

    Authorization ‒ ϩʔϧϕʔεͷΞΫηείϯτϩʔϧ ‒ ΠϯσοΫε͝ͱɺΞΫγϣϯ͝ͱͷઃఆ͕Մೳ ‒ υΩϡϝϯτɾϑΟʔϧυ͝ͱͷઃఆ΋Մೳʹ • ηΩϡΞͳ௨৴ ‒ ElasticsearchϊʔυؒͷSSL/TLSɺIPϑΟϧλϦϯά • ؂ࠪϩά 65

66. Alerting

67. X-Pack : Alertingͷಛ௃ • ΫΤϦʹΑΔWatch ‒ ElasticsearchͷΫΤϦΛར༻ͯ͠σʔλͷ؂ࢹ • ৚݅ͷઃఆ ‒

    ΞΫγϣϯΛ࣮ߦ͢Δ͔Ͳ͏͔ͷઃఆ • εέδϡʔϧ ‒ ΫΤϦΛ࣮ߦ͠ɺ৚݅ΛνΣοΫ͢Δස౓ͷࢦఆ • ΞΫγϣϯͷఆٛ ‒ ϝʔϧͷૹ৴ɺଞγεςϜ΁ͷσʔλૹ৴ͳͲͷಈ࡞Λઃఆ • ཤྺͷอଘ 67

68. Graph

69. Graphͷಛ௃ • σʔλؒͷͭͳ͕ΓΛ୳ࡧ͢ΔϓϥάΠϯ • KibanaϓϥάΠϯʹΑΓGUIΛར༻ͯ͠୳ࡧՄೳ 69

70. Prelert

71. σʔλ͔Β༗ҙٛͳ৘ใΛݟ͚ͭΔํ๏
    Search Aggregations Visualization Machine Learning

72. 1SFMFSUͷςΫϊϩδʔ
    σʔλʹજΉߦಈϞσϧΛ
 ࣗಈతʹڭࢣͳֶ͠श ݱࡏͷߦಈ͕༧ଌϞσϧͱ
 ݦஶʹҟͳΔ৔߹ʹ௨஌

73. 73 σϞ Demo

74. ࢀߟαΠτ • Ϣʔεέʔε • https://www.elastic.co/use-cases • DiscussʢWebϑΥʔϥϜʣ • https://discuss.elastic.co •

    Elastic{ON}ͷϏσΦͱࢿྉ • https://www.elastic.co/elasticon/videos • αϙʔτϝχϡʔ • https://www.elastic.co/subscriptions 74

75. 75 March 7-9, 2017 • Pier 48 • San Francisco,

    CA • 2,500 attendees 3rd Annual Elastic User Conference Topics • Latest Roadmap • Ask Me Anything Booth • 70+ Sessions • 76 Demo Hours

76. Ξϯέʔτ΁ͷճ౴Λ͓ئ͍͠·͢ bit.ly/bigdata-tokyo-elastic

77. Thanks for listening! Q & A We’re hiring! https://www.elastic.co/about/careers/ We’re

    helping! https://www.elastic.co/subscriptions http://training.elastic.co