Elastic Stackを利用してデータから様々な気づきを見つける

Slide 1

Slide 1 text

‹#› 2017/02/07 Evangelist at Elastic Jun Ohtani @johtani Elastic StackΛར༻ͯ͠ σʔλ͔Β༷ʑͳؾ͖ͮΛݟ͚ͭΔ

Slide 2

Slide 2 text

‹#›

Slide 3

Slide 3 text

ΞδΣϯμ • ؾ͖ͮΛݟ͚ͭΔͱ͸ʁ • Ϣʔεέʔεͷ঺հ • Elastic stack঺հ • BeatsɺLogstashɺElasticsearchɺKibanaɺX-Pack • σϞ 3

Slide 4

Slide 4 text

about • Me, Jun Ohtani / Technical Advocate ‒ lucene-gosenίϛολʔ ‒ ElasticSearch Server೔ຊޠ൛ͷ຋༁ ‒ http://blog.johtani.info  • Elasticsearch, founded in 2012 ‒ Products: Elasticsearch, Logstash, Kibana, Beats   X-Pack, Elastic Cloud  Professional services: Support & development subscriptions ‒ Trainings & Consulting 4

Slide 5

Slide 5 text

༷ʑͳϢʔεέʔε 5 ؾ͖ͮΛݟ͚ͭΔ ͱ͸ʁ

Slide 6

Slide 6 text

Search and analytics, it all started here More than 60% of our customers have a search or analytics use case

Slide 7

Slide 7 text

Slide 8

Slide 8 text

Slide 9

Slide 9 text

Logs Logs Logs,   many devices,  many systems More than 40% of our  customers use our products  for operational log analysis

Slide 10

Slide 10 text

We collect more than 1.2 TB logs every day from our infrastructure, web servers, and applications. 10

Slide 11

Slide 11 text

11 We handle more than 3 Billion daily events while meeting our all of our data security requirements.

Slide 12

Slide 12 text

Sniff sniff sniff,  ﬁnd the bad actors  in your data 200% YoY growth in security use cases with our products

Slide 13

Slide 13 text

We analyze piles of data: 13B AMP queries/day 600B emails/day 16B web requests/day 13

Slide 14

Slide 14 text

14 We mine and analyze 4 billion events every day to detect security hacks and threats. 1

Slide 15

Slide 15 text

The Elastic Stack:   A foundation to solve many use cases 75% of our customers use our products for more than one use case SEARCH SECURIT CUSTOM APPS METRICS OPERATIONAL  ANALYTICS LOG ANALYSIS

Slide 16

Slide 16 text

Operational analytics Flight telemetry analysis Anomaly resolution Internal search engine 16

Slide 17

Slide 17 text

17 Enterprise search Intranet search Real-time log analytics Legal contract repository Trade tracking application HR recruiting application

Slide 18

Slide 18 text

18 ElasticελοΫ

Slide 19

Slide 19 text

ElasticελοΫʢOpen Sourceʣ 19 Kibana Elasticsearch Logstash Beats

Slide 20

Slide 20 text

ElasticελοΫ 20 Elastic Cloud X-Pack Kibana Elasticsearch ! " Logstash Beats +

Slide 21

Slide 21 text

Ingest

Slide 22

Slide 22 text

22 Logstash

Slide 23

Slide 23 text

Logstash in 10 seconds • ϩάɾσʔλͷऩूɾ؅ཧ • ऩूɺύʔεɾՃ޻ɺૹग़ • ΦʔϓϯιʔεɿApache License 2.0 • Ruby app (JRuby) 23

Slide 24

Slide 24 text

Logstash architecture 24 Input Output Filter ? ? collect and split alter and enrich store and visualize

Slide 25

Slide 25 text

ઃఆ 25 input { … } filter { … } output { … }

Slide 26

Slide 26 text

ઃఆɿinput 26 input { file { path => “/Users/johtani/sample/*_log" start_position => "beginning" } }

Slide 27

Slide 27 text

1ߦ1σʔλ 189.120.xx.xx - - [02/Dec/2014:12:18:29 +0900] "GET /manager/html HTTP/ 1.1" 404 274 "-" "Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0" 27

Slide 28

Slide 28 text

ઃఆɿfilter 28 filter { grok { match => { "message" => "%{COMBINEDAPACHELOG}" } break_on_match => false } date { match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"] locale => en } geoip { source => ["clientip"] } useragent { source => "agent" target => "useragent" } }

Slide 29

Slide 29 text

ύʔε 29 189.120.xx.xx - - [02/Dec/2014:12:18:29 +0900] "GET /manager/html HTTP/1.1" 404 274 "-" "Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0" {… "@timestamp": "2015-04-10T09:07:49.325Z", "clientip": "189.120.xx.xx", "ident": "-", "auth": "-", "timestamp": "02/Dec/2014:12:18:29 +0900", "verb": "GET", "request": "/manager/html", … "agent": "\"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/

Slide 30

Slide 30 text

ઃఆɿfilter 30 filter { grok { match => { "message" => "%{COMBINEDAPACHELOG}" } break_on_match => false } date { match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"] locale => en } geoip { source => ["clientip"] } useragent { source => "agent" target => "useragent" } }

Slide 31

Slide 31 text

IP͔ΒҢ౓ܦ౓ͳͲ෇༩ 31 "clientip": "189.120.xx.xx", "clientip": "189.120.xx.xx", "geoip": { "ip": “189.120.xxx.xxx”, … "country_name": "Brazil", "continent_code": "SA", "region_name": "27", "city_name": "São Paulo", "latitude":

Slide 32

Slide 32 text

ઃఆɿoutput 32 output { elasticsearch { hosts => ["localhost"] index => “demo_access_log-%{+YYYY.MM.dd}” } }

Slide 33

Slide 33 text

ܰྔσʔλγούʔ 33 Beats

Slide 34

Slide 34 text

To tail a File ﬁlebeat

Slide 35

Slide 35 text

To tail a File ﬁlebeat

Slide 36

Slide 36 text

Capture the Packet Packetbeat

Slide 37

Slide 37 text

Capture the Packet Packetbeat

Slide 38

Slide 38 text

Welcome to 1998 winlogbeat

Slide 39

Slide 39 text

Now winlogbeat

Slide 40

Slide 40 text

Store, Search & Analytics

Slide 41

Slide 41 text

41 Elasticsearch

Slide 42

Slide 42 text

ݕࡧͱͯ͠ͷ  Elasticsearch

Slide 43

Slide 43 text

Elasticsearchͱ͸ʁ

Slide 44

Slide 44 text

ϑϦʔϫʔυݕࡧ 44

Slide 45

Slide 45 text

ߜΓࠐΈ 45

Slide 46

Slide 46 text

ϋΠϥΠτ 46

Slide 47

Slide 47 text

ιʔτ 47

Slide 48

Slide 48 text

ϖʔδϯά 48

Slide 49

Slide 49 text

ूܭ 49

Slide 50

Slide 50 text

αδΣετ 50

Slide 51

Slide 51 text

Elasticsearch in 10 seconds • εΩʔϚϑϦʔɺ෼ࢄυΩϡϝϯτετΞɺREST & JSON • Φʔϓϯιʔε: Apache License 2.0 • ઃఆͳ͠Ͱ؆୯ʹࢼ͢͜ͱ͕Մೳ • JavaͰ࣮૷ɻ֦ு΋༰қ 51

Slide 52

Slide 52 text

ղੳͱͯ͠ͷ  Elasticsearch

Slide 53

Slide 53 text

aggregation

Slide 54

Slide 54 text

Aggregationͱ͸ • 1.0͔Βಋೖ • FacetΑΓ΋ڧྗͳूܭͳͲ͕Մೳ • ֊૚తͳूܭɺάϧʔϓԽ  ಈతͳूܭɺάϧʔϓԽ • େ͖͘2छྨ • BucketɹυΩϡϝϯτΛ஋͝ͱʹ݁ՌΛάϧʔϐϯά • Metricɹ υΩϡϝϯτͷ࣋ͭ஋Λूܭ 54

Slide 55

Slide 55 text

ྫɿݴޠ͓Αͼ஍Ҭͷूܭ 55 curl -XGET twitter-2014.08.22/_search -d ' { "aggs": { "lang": { "terms": {"field": "lang" }, "aggs": { "place": { "terms": { "field": “place.full_name", "size": 10 } } } } } }

Slide 56

Slide 56 text

ྫɿݴޠ͓Αͼ஍Ҭͷूܭ 56 "aggregations": { "lang": { "buckets": [{…}, { "key": "ja", "doc_count": 980145, "place": { "buckets": [ { "key": "ژ౎ࢢ෬ݟ۠, ژ౎", "doc_count":252 }, { "key": "ઍ୅ా۠, ౦ژ", "doc_count": 39 },…

Slide 57

Slide 57 text

elasticsearch-hadoop 57 - •  D E H •  PD ecd ER •  g D •  CH •  Ca M DMS D FERC

Slide 58

Slide 58 text

The Window into the Elastic Stack

Slide 59

Slide 59 text

59 KibanaͰՄࢹԽ

Slide 60

Slide 60 text

Kibana 5 • ElasticsearchͷσʔλΛՄࢹԽ • Node.js server & JavaScript • Apache License 2.0 • Elastic Stackͷ૭ͷ໾ׂ • ༷ʑͳGUIΛPluginͱ͍ͯެ։ • MarvelɺSenseɺTimelionͳͲ 60

Slide 61

Slide 61 text

Kibana 5 61

Slide 62

Slide 62 text

No content

Slide 63

Slide 63 text

X-Pack 5.0: Extending the Elastic Stack

Slide 64

Slide 64 text

Security

Slide 65

Slide 65 text

X-Pack : Securityͷಛ௃ • User Authentication ‒ LDAP/Active Directory/ϑΝΠϧϕʔε • Authorization ‒ ϩʔϧϕʔεͷΞΫηείϯτϩʔϧ ‒ ΠϯσοΫε͝ͱɺΞΫγϣϯ͝ͱͷઃఆ͕Մೳ ‒ υΩϡϝϯτɾϑΟʔϧυ͝ͱͷઃఆ΋Մೳʹ • ηΩϡΞͳ௨৴ ‒ ElasticsearchϊʔυؒͷSSL/TLSɺIPϑΟϧλϦϯά • ؂ࠪϩά 65

Slide 66

Slide 66 text

Alerting

Slide 67

Slide 67 text

X-Pack : Alertingͷಛ௃ • ΫΤϦʹΑΔWatch ‒ ElasticsearchͷΫΤϦΛར༻ͯ͠σʔλͷ؂ࢹ • ৚݅ͷઃఆ ‒ ΞΫγϣϯΛ࣮ߦ͢Δ͔Ͳ͏͔ͷઃఆ • εέδϡʔϧ ‒ ΫΤϦΛ࣮ߦ͠ɺ৚݅ΛνΣοΫ͢Δස౓ͷࢦఆ • ΞΫγϣϯͷఆٛ ‒ ϝʔϧͷૹ৴ɺଞγεςϜ΁ͷσʔλૹ৴ͳͲͷಈ࡞Λઃఆ • ཤྺͷอଘ 67

Slide 68

Slide 68 text

Graph

Slide 69

Slide 69 text

Graphͷಛ௃ • σʔλؒͷͭͳ͕ΓΛ୳ࡧ͢ΔϓϥάΠϯ • KibanaϓϥάΠϯʹΑΓGUIΛར༻ͯ͠୳ࡧՄೳ 69

Slide 70

Slide 70 text

Prelert

Slide 71

Slide 71 text

σʔλ͔Β༗ҙٛͳ৘ใΛݟ͚ͭΔํ๏ Search Aggregations Visualization Machine Learning

Slide 72

Slide 72 text

1SFMFSUͷςΫϊϩδʔ σʔλʹજΉߦಈϞσϧΛ  ࣗಈతʹڭࢣͳֶ͠श ݱࡏͷߦಈ͕༧ଌϞσϧͱ  ݦஶʹҟͳΔ৔߹ʹ௨஌

Slide 73

Slide 73 text

73 σϞ Demo

Slide 74

Slide 74 text

ࢀߟαΠτ • Ϣʔεέʔε • https://www.elastic.co/use-cases • DiscussʢWebϑΥʔϥϜʣ • https://discuss.elastic.co • Elastic{ON}ͷϏσΦͱࢿྉ • https://www.elastic.co/elasticon/videos • αϙʔτϝχϡʔ • https://www.elastic.co/subscriptions 74

Slide 75

Slide 75 text

75 March 7-9, 2017 • Pier 48 • San Francisco, CA • 2,500 attendees 3rd Annual Elastic User Conference Topics • Latest Roadmap • Ask Me Anything Booth • 70+ Sessions • 76 Demo Hours

Slide 76

Slide 76 text

Ξϯέʔτ΁ͷճ౴Λ͓ئ͍͠·͢ bit.ly/bigdata-tokyo-elastic

Slide 77

Slide 77 text

Thanks for listening! Q & A We’re hiring! https://www.elastic.co/about/careers/ We’re helping! https://www.elastic.co/subscriptions http://training.elastic.co