Intro Elastic Stack at Telemetry WG

Slide 1

Slide 1 text

!1 Jun Ohtani 2018/06/29 at TelemetryWG @johtani Elastic stackͷ঺հ

Slide 2

Slide 2 text

‹#›

Slide 3

Slide 3 text

about • Me, Jun Ohtani / Technical Advocate ‒ lucene-gosenίϛολʔ ‒ ElasticSearch Server೔ຊޠ൛ͷ຋༁ ‒ http://blog.johtani.info  • Elasticsearch, founded in 2012 ‒ Products: Elasticsearch, Logstash, Kibana, Beats   X-Pack, Elastic Cloud  Professional services: Support & development subscriptions ‒ Trainings, Consulting, SaaS !3

Slide 4

Slide 4 text

4 ElasticελοΫ

Slide 5

Slide 5 text

!5 Elastic Stack Φʔϓϯιʔε ΛϦϦʔε

Slide 6

Slide 6 text

Πϯετʔϧ͸؆୯ &MBTUJD4UBDLΛ֦ு αϒεΫϦϓγϣϯʹؚΉ X-Pack !6 Security Alerting Monitoring Reporting Graph Machine Learning

Slide 7

Slide 7 text

ϝτϦΫε/ϩάղੳΛ ࢝ΊͯΈΑ͏

Slide 8

Slide 8 text

ϝτϦΫεɾϩάͷ෼ੳʢ؆қ൛ʣ !8 σʔλ Import Parse/  Store/Search Visualize

Slide 9

Slide 9 text

ܰྔσʔλγούʔ 9 Beats

Slide 10

Slide 10 text

10 Beats ܰྔσʔλγούʔ ιʔε͔ΒσʔλΛసૹ సૹ͠Elasticsearchʹू໿ ม׵ͱύʔεͷͨΊ Logstashʹసૹ Elastic Cloudʹసૹ Libbeat: ΧελϜbeatsͷͨ ΊͷAPIϑϨʔϜϫʔΫ 30Ҏ্ͷίϛϡχςΟbeats

Slide 11

Slide 11 text

11 FILEBEAT ϩάϑΝΠϧ METRICBEAT ϝτϦοΫ৘ใ PACKETBEAT ωοτϫʔΫ WINGLOGBEAT WindowΠϕϯτ ͞Βʹ30Λ௒͑ΔίϛϡχςΟ Beats͕͋Γɺ૿Ճத Apachebeat, dockbeat, httpbeat, mysqlbeat, nginxbeat, redis beats, twitterbeat, and more

Slide 12

Slide 12 text

Collect system and application metrics Metricbeat

Slide 13

Slide 13 text

lots of modules Metricbeat

Slide 14

Slide 14 text

tail log from ﬁle Filebeat

Slide 15

Slide 15 text

many modules Filebeat

Slide 16

Slide 16 text

Capture the Packet Packetbeat

Slide 17

Slide 17 text

Capture the Packet Packetbeat

Slide 18

Slide 18 text

Welcome to 1998 winlogbeat

Slide 19

Slide 19 text

Now winlogbeat

Slide 20

Slide 20 text

20 Elasticsearch

Slide 21

Slide 21 text

ݕࡧͱͯ͠ͷ  Elasticsearch

Slide 22

Slide 22 text

؆୯ͳCRUD

Slide 23

Slide 23 text

σʔλొ࿥ 23 curl -XPUT localhost:9200/books/book/1 -d ' { "title" : "Elasticsearch - The definitive guide", "authors" : "Clinton Gormley", "started" : "2013-02-04", "pages" : 230 }'

Slide 24

Slide 24 text

σʔλߋ৽ 24 curl -XPUT localhost:9200/books/book/1 -d ' { "title" : "Elasticsearch - The definitive guide", "authors" : [ "Clinton Gormley", "Zachary Tong" ], "started" : "2013-02-04", "pages" : 230 }'

Slide 25

Slide 25 text

σʔλ࡟আ !25 curl -X DELETE localhost:9200/books/book/1 σʔλͷऔಘ curl —X GET localhost:9200/books/book/1 curl —X GET localhost:9200/books/book/1/_source

Slide 26

Slide 26 text

ݕࡧ !26 curl -XGET localhost:9200/books/_search?q=elasticsearch { "took" : 2, "timed_out" : false, "_shards" : { "total" : 5, "successful" : 5, "failed" : 0 }, "hits" : { "total" : 1, "max_score" : 0.076713204, "hits" : [ { "_index" : “books", "_type" : “book", "_id" : "1", "_score" : 0.076713204, "_source" : { "title" : "Elasticsearch - The definitive guide", "authors" : [ "Clinton Gormley", "Zachary Tong" ], "started" : “2013-02-04", "pages" : 230 } } ]

Slide 27

Slide 27 text

ݕࡧ - Query DSL !27 curl -XGET ‘localhost:9200/books/book/_search' -d '{ "query": { "filtered" : { "query" : { "match": { "text" : { "query" : “To Be Or Not To Be", "cutoff_frequency" : 0.01 } } }, "filter" : { "range": { "price": { "gte": 20.0 "lte": 50.0

Slide 28

Slide 28 text

෼ࢄߏ੒ɺ  εέʔϧ

Slide 29

Slide 29 text

Basic terms • ΠϯσοΫε ‒ σʔλͷ࿦ཧతͳू߹ɻ  RDBͷσʔλϕʔεͷΑ͏ͳ΋ͷLogical • ϨϓϦέʔγϣϯ • ಡΈࠐΈͷεέʔϥϏϦςΟ޲্ • SPOFͷղফ • γϟʔσΟϯά • ෳ਺Ϛγϯ΁σʔλΛ෼ׂ  ॻ͖ࠐΈͷεέʔϥϏϦςΟ޲্  σʔλϑϩʔ੍ޚ !29

Slide 30

Slide 30 text

γϟʔυͱϨϓϦΧ !30 node 1 orders products 1 4 1 2 2 3 curl -X PUT localhost:9200/orders -d '{ "settings.index.number_of_shards" : 4 "settings.index.number_of_replicas" : 1 }' curl -X PUT localhost:9200/products -d '{ "settings.index.number_of_shards" : 2 "settings.index.number_of_replicas" : 0 }'

Slide 31

Slide 31 text

γϟʔυͱϨϓϦΧ !31 node 1 orders products 1 4 1 node 2 orders products 2 2 3 4 1 2 3

Slide 32

Slide 32 text

ࣗಈతͳ෼ࢄ !32 node 1 orders products 2 1 4 1 node 2 orders products 2 2 node 3 orders products 3 4 1 3

Slide 33

Slide 33 text

ͦͷଞͷػೳ

Slide 34

Slide 34 text

elasticsearch ͞·͟·ͳܗࣜͷσʔλͰ GeoݕࡧՄೳ    Ң౓ܦ౓ɺGeoHashɺ GeoShape… GEO

Slide 35

Slide 35 text

Ecosystem • Plugins ‒ ϓϥάΠϯʹΑΔػೳͷ௥Ճ • ΫϥΠΞϯτϥΠϒϥϦ • Java, Ruby, python, php, perl, javascript, .NET • Scala, clojure, go !35

Slide 36

Slide 36 text

Elasticsearch - The Definitive guide    http://www.elastic.co/guide/en/ elasticsearch/guide/current/index.html 36 ৄ͘͠஌Γ͍ͨํ͸

Slide 37

Slide 37 text

37 KibanaͰՄࢹԽ

Slide 38

Slide 38 text

Kibana 5 • ElasticsearchͷσʔλΛՄࢹԽ • Node.js server & JavaScript • Apache License 2.0 • Elastic Stackͷ૭ͷ໾ׂ • ༷ʑͳGUIΛPluginͱ͍ͯެ։ • MarvelɺSenseɺTimelionͳͲ !38

Slide 39

Slide 39 text

Kibana 5 39

Slide 40

Slide 40 text

Combining Search and Analytics !40

Slide 41

Slide 41 text

σϞ for Kibana5 Access Log 41

Slide 42

Slide 42 text

ຊ֨తʹղੳΛߦ͏ʹ͸ʁ

Slide 43

Slide 43 text

No content

Slide 44

Slide 44 text

44 Logstash

Slide 45

Slide 45 text

Logstash in 10 seconds • ϩάɾσʔλͷऩूɾ؅ཧ • ऩूɺύʔεɾՃ޻ɺૹग़ • ΦʔϓϯιʔεɿApache License 2.0 • Ruby app (JRuby) !45

Slide 46

Slide 46 text

Logstash architecture !46 Input Output Filter ? ? collect and split alter and enrich store and visualize

Slide 47

Slide 47 text

ઃఆ 47 input { … } filter { … } output { … }

Slide 48

Slide 48 text

ઃఆɿinput 48 input { file { path => “/Users/johtani/sample/*_log" start_position => "beginning" } }

Slide 49

Slide 49 text

1ߦ1σʔλ 189.120.xx.xx - - [02/Dec/2014:12:18:29 +0900] "GET /manager/html HTTP/ 1.1" 404 274 "-" "Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0" 49

Slide 50

Slide 50 text

ઃఆɿfilter 50 filter { grok { match => { "message" => "%{COMBINEDAPACHELOG}" } break_on_match => false } date { match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"] locale => en } geoip { source => ["clientip"] } useragent { source => "agent" target => "useragent" } }

Slide 51

Slide 51 text

ύʔε !51 189.120.xx.xx - - [02/Dec/2014:12:18:29 +0900] "GET /manager/html HTTP/1.1" 404 274 "-" "Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0" {… "@timestamp": "2015-04-10T09:07:49.325Z", "clientip": "189.120.xx.xx", "ident": "-", "auth": "-", "timestamp": "02/Dec/2014:12:18:29 +0900", "verb": "GET", "request": "/manager/html", … "agent": "\"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/

Slide 52

Slide 52 text

ઃఆɿfilter !52 filter { grok { match => { "message" => "%{COMBINEDAPACHELOG}" } break_on_match => false } date { match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"] locale => en } geoip { source => ["clientip"] } useragent { source => "agent" target => "useragent" } }

Slide 53

Slide 53 text

೔෇ͷύʔε 53 {… "@timestamp": "2015-04-10T09:07:49.325Z", … "timestamp": "02/Dec/2014:12:18:29 +0900", … } {… "@timestamp": "2014-12-02T03:18:29.000Z", … "timestamp": "02/Dec/2014:12:18:29 +0900", … }

Slide 54

Slide 54 text

ઃఆɿfilter !54 filter { grok { match => { "message" => "%{COMBINEDAPACHELOG}" } break_on_match => false } date { match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"] locale => en } geoip { source => ["clientip"] } useragent { source => "agent" target => "useragent" } }

Slide 55

Slide 55 text

IP͔ΒҢ౓ܦ౓ͳͲ෇༩ 55 "clientip": "189.120.xx.xx", "clientip": "189.120.xx.xx", "geoip": { "ip": “189.120.xxx.xxx”, … "country_name": "Brazil", "continent_code": "SA", "region_name": "27", "city_name": "São Paulo", "latitude":

Slide 56

Slide 56 text

ઃఆɿfilter !56 filter { grok { match => { "message" => "%{COMBINEDAPACHELOG}" } break_on_match => false } date { match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"] locale => en } geoip { source => ["clientip"] } useragent { source => "agent" target => "useragent" } }

Slide 57

Slide 57 text

ϢʔβΤʔδΣϯτͷύʔε 57 "agent": "\"Mozilla/5.0 (Windows NT 5.1; rv: 5.0) Gecko/20100101 Firefox/5.0\"" "agent": "\"Mozilla/5.0 (Windows NT 5.1; rv: 5.0) Gecko/20100101 Firefox/5.0\"" "useragent": { "name": "Firefox", "os": "Windows XP", "os_name": "Windows XP", "device": "Other", "major": "5", "minor": "0"

Slide 58

Slide 58 text

ઃఆɿoutput 58 output { elasticsearch { hosts => ["localhost"] index => “demo_access_log-%{+YYYY.MM.dd}” } }

Slide 59

Slide 59 text

ࢀߟจݙ • Elasticsearch - The Definitive guide ‒ http://www.elastic.co/guide/en/elasticsearch/guide/current/index.html • ॻ੶ʢ೔ຊޠʣ ‒ σʔλ෼ੳج൫ߏஙೖ໳ ‒ Elasticsearch࣮ફΨΠυ !59

Slide 60

Slide 60 text

ࢀߟαΠτ • Ϣʔεέʔε • https://www.elastic.co/use-cases • DiscussʢWebϑΥʔϥϜʣ • https://discuss.elastic.co • Elastic{ON}ͷϏσΦͱࢿྉ • https://www.elastic.co/elasticon/videos • αϙʔτϝχϡʔ • https://www.elastic.co/subscriptions !60

Slide 61

Slide 61 text

Thanks for listening! Q & A We’re hiring! https://www.elastic.co/about/careers/ We’re helping! https://www.elastic.co/subscriptions http://training.elastic.co

Slide 62

Slide 62 text

QAͰ঺հͨ͠ػೳ

Slide 63

Slide 63 text

!63 Filebeatͷೖྗ͕ଟ༷ʹ • TCP Input ‒ SSL/TLSΛαϙʔτ • UDP Input • Syslog Input ‒ BSD RFC3164Λαϙʔτ ‒ ϓϩτίϧ͸TCPͱUDP filebeat.inputs:  - type: tcp  max_message_size: 10MiB  host: "localhost:9000" filebeat.inputs:  - type: udp  max_message_size: 10KiB  host: "localhost:8080" filebeat.inputs:  - type: syslog  protocol.tcp:  host: "localhost:9000" filebeat.yml

Slide 64

Slide 64 text

!64 Data Rollups ● API for creating an Elasticsearch process to periodically store aggregate statistics ● Primary benefit is space savings ○ Faster queries ○ Potentially less nodes to manage ○ Smaller snapshots ○ Longer retention times ○ etc. ● Query rolled up data and “live” data together in a single query. Rollups API (6.3 - Experimental) ● ఆظతʹ౷ܭσʔλΛू໿ͯ͠อଘ͢ΔElasticsearchͷJobΛొ࿥ ● ओͳར఺͸༰ྔͷ࡟ݮ ● σʔλ͕গͳ͘ͳΔͨΊ ○ Query͕ΑΓߴ଎ʹ ○ গͳ͍ϊʔυͰσʔλΛ؅ཧ ○ Snapshot͕ΑΓখ͘͞ ○ σʔλͷอ࣋ظ͕ؒΑΓ௕͘ ● 1ͭͷΫΤϦͰϩʔϧΞοϓͨ͠σʔλͱͯ͠ͳ͍σʔλΛ໰͍߹Θͤ ༰ྔ͕ɻɻɻ X-Pack feature (Basic, free)

Slide 65

Slide 65 text

!65 Raw Minute Hour Day Docs: 9,041,000 1,448,285 49,554 8,447 Size: 2.23gb 1.25gb 48.40mb 9.10mb Docs % : -83.98% -99.45% -99.91% Size %: -43.68% -97.84% -99.59% (avg ~200 docs per minute, 32 days of data, single host) (20 grouping fields, 62 numerics @ min/max/avg == 186 metrics) Rolling up Metricbeat data ༰ྔ࡟ݮͷҰྫ MetricbeatͷϩʔϧΞοϓ (ฏۉ ~200 docs/෼ɺ32೔ؒɺ1αʔόʔ) (20ݸͷάϧʔϓϑΟʔϧυɺ62ݸͷ਺஋ @ min/max/avg == 186 metrics) X-Pack feature (Basic, free)