Intro Elastic Stack at Telemetry WG

Intro Elastic Stack at Telemetry WG

2018/06/29にTelemetryWGで使用したスライド。

657aeeff3fc467567dacebf8a1ea0b23?s=128

Jun Ohtani

July 05, 2018
Tweet

Transcript

  1. !1 Jun Ohtani 2018/06/29 at TelemetryWG @johtani Elastic stackͷ঺հ

  2. ‹#›

  3. about • Me, Jun Ohtani / Technical Advocate ‒ lucene-gosenίϛολʔ

    ‒ ElasticSearch Server೔ຊޠ൛ͷ຋༁ ‒ http://blog.johtani.info
 • Elasticsearch, founded in 2012 ‒ Products: Elasticsearch, Logstash, Kibana, Beats 
 X-Pack, Elastic Cloud
 Professional services: Support & development subscriptions ‒ Trainings, Consulting, SaaS !3
  4. 4 ElasticελοΫ

  5. !5 Elastic Stack Φʔϓϯιʔε ΛϦϦʔε

  6. Πϯετʔϧ͸؆୯ &MBTUJD4UBDLΛ֦ு αϒεΫϦϓγϣϯʹؚΉ X-Pack !6 Security Alerting Monitoring Reporting Graph

    Machine Learning
  7. ϝτϦΫε/ϩάղੳΛ ࢝ΊͯΈΑ͏

  8. ϝτϦΫεɾϩάͷ෼ੳʢ؆қ൛ʣ !8 σʔλ Import Parse/
 Store/Search Visualize

  9. ܰྔσʔλγούʔ 9 Beats

  10. 10 Beats ܰྔσʔλγούʔ ιʔε͔ΒσʔλΛసૹ సૹ͠Elasticsearchʹू໿ ม׵ͱύʔεͷͨΊ Logstashʹసૹ Elastic Cloudʹసૹ Libbeat:

    ΧελϜbeatsͷͨ ΊͷAPIϑϨʔϜϫʔΫ 30Ҏ্ͷίϛϡχςΟbeats
  11. 11 FILEBEAT ϩάϑΝΠϧ METRICBEAT ϝτϦοΫ৘ใ PACKETBEAT ωοτϫʔΫ WINGLOGBEAT WindowΠϕϯτ ͞Βʹ30Λ௒͑ΔίϛϡχςΟ

    Beats͕͋Γɺ૿Ճத Apachebeat, dockbeat, httpbeat, mysqlbeat, nginxbeat, redis beats, twitterbeat, and more
  12. Collect system and application metrics Metricbeat

  13. lots of modules Metricbeat

  14. tail log from file Filebeat

  15. many modules Filebeat

  16. Capture the Packet Packetbeat

  17. Capture the Packet Packetbeat

  18. Welcome to 1998 winlogbeat

  19. Now winlogbeat

  20. 20 Elasticsearch

  21. ݕࡧͱͯ͠ͷ
 Elasticsearch

  22. ؆୯ͳCRUD

  23. σʔλొ࿥ 23 curl -XPUT localhost:9200/books/book/1 -d ' { "title" :

    "Elasticsearch - The definitive guide", "authors" : "Clinton Gormley", "started" : "2013-02-04", "pages" : 230 }'
  24. σʔλߋ৽ 24 curl -XPUT localhost:9200/books/book/1 -d ' { "title" :

    "Elasticsearch - The definitive guide", "authors" : [ "Clinton Gormley", "Zachary Tong" ], "started" : "2013-02-04", "pages" : 230 }'
  25. σʔλ࡟আ !25 curl -X DELETE localhost:9200/books/book/1 σʔλͷऔಘ curl —X GET

    localhost:9200/books/book/1 curl —X GET localhost:9200/books/book/1/_source
  26. ݕࡧ !26 curl -XGET localhost:9200/books/_search?q=elasticsearch { "took" : 2, "timed_out"

    : false, "_shards" : { "total" : 5, "successful" : 5, "failed" : 0 }, "hits" : { "total" : 1, "max_score" : 0.076713204, "hits" : [ { "_index" : “books", "_type" : “book", "_id" : "1", "_score" : 0.076713204, "_source" : { "title" : "Elasticsearch - The definitive guide", "authors" : [ "Clinton Gormley", "Zachary Tong" ], "started" : “2013-02-04", "pages" : 230 } } ]
  27. ݕࡧ - Query DSL !27 curl -XGET ‘localhost:9200/books/book/_search' -d '{

    "query": { "filtered" : { "query" : { "match": { "text" : { "query" : “To Be Or Not To Be", "cutoff_frequency" : 0.01 } } }, "filter" : { "range": { "price": { "gte": 20.0 "lte": 50.0
  28. ෼ࢄߏ੒ɺ
 εέʔϧ

  29. Basic terms • ΠϯσοΫε ‒ σʔλͷ࿦ཧతͳू߹ɻ
 RDBͷσʔλϕʔεͷΑ͏ͳ΋ͷLogical • ϨϓϦέʔγϣϯ •

    ಡΈࠐΈͷεέʔϥϏϦςΟ޲্ • SPOFͷղফ • γϟʔσΟϯά • ෳ਺Ϛγϯ΁σʔλΛ෼ׂ
 ॻ͖ࠐΈͷεέʔϥϏϦςΟ޲্
 σʔλϑϩʔ੍ޚ !29
  30. γϟʔυͱϨϓϦΧ !30 node 1 orders products 1 4 1 2

    2 3 curl -X PUT localhost:9200/orders -d '{ "settings.index.number_of_shards" : 4 "settings.index.number_of_replicas" : 1 }' curl -X PUT localhost:9200/products -d '{ "settings.index.number_of_shards" : 2 "settings.index.number_of_replicas" : 0 }'
  31. γϟʔυͱϨϓϦΧ !31 node 1 orders products 1 4 1 node

    2 orders products 2 2 3 4 1 2 3
  32. ࣗಈతͳ෼ࢄ !32 node 1 orders products 2 1 4 1

    node 2 orders products 2 2 node 3 orders products 3 4 1 3
  33. ͦͷଞͷػೳ

  34. elasticsearch ͞·͟·ͳܗࣜͷσʔλͰ GeoݕࡧՄೳ
 
 Ң౓ܦ౓ɺGeoHashɺ GeoShape… GEO

  35. Ecosystem • Plugins ‒ ϓϥάΠϯʹΑΔػೳͷ௥Ճ • ΫϥΠΞϯτϥΠϒϥϦ • Java, Ruby,

    python, php, perl, javascript, .NET • Scala, clojure, go !35
  36. Elasticsearch - The Definitive guide
 
 http://www.elastic.co/guide/en/ elasticsearch/guide/current/index.html 36 ৄ͘͠஌Γ͍ͨํ͸

  37. 37 KibanaͰՄࢹԽ

  38. Kibana 5 • ElasticsearchͷσʔλΛՄࢹԽ • Node.js server & JavaScript •

    Apache License 2.0 • Elastic Stackͷ૭ͷ໾ׂ • ༷ʑͳGUIΛPluginͱ͍ͯެ։ • MarvelɺSenseɺTimelionͳͲ !38
  39. Kibana 5 39

  40. Combining Search and Analytics !40

  41. σϞ for Kibana5 Access Log 41

  42. ຊ֨తʹղੳΛߦ͏ʹ͸ʁ

  43. None
  44. 44 Logstash

  45. Logstash in 10 seconds • ϩάɾσʔλͷऩूɾ؅ཧ • ऩूɺύʔεɾՃ޻ɺૹग़ • ΦʔϓϯιʔεɿApache

    License 2.0 • Ruby app (JRuby) !45
  46. Logstash architecture !46 Input Output Filter ? ? collect and

    split alter and enrich store and visualize
  47. ઃఆ 47 input { … } filter { … }

    output { … }
  48. ઃఆɿinput 48 input { file { path => “/Users/johtani/sample/*_log" start_position

    => "beginning" } }
  49. 1ߦ1σʔλ 189.120.xx.xx - - [02/Dec/2014:12:18:29 +0900] "GET /manager/html HTTP/ 1.1"

    404 274 "-" "Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0" 49
  50. ઃఆɿfilter 50 filter { grok { match => { "message"

    => "%{COMBINEDAPACHELOG}" } break_on_match => false } date { match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"] locale => en } geoip { source => ["clientip"] } useragent { source => "agent" target => "useragent" } }
  51. ύʔε !51 189.120.xx.xx - - [02/Dec/2014:12:18:29 +0900] "GET /manager/html HTTP/1.1"

    404 274 "-" "Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0" {… "@timestamp": "2015-04-10T09:07:49.325Z", "clientip": "189.120.xx.xx", "ident": "-", "auth": "-", "timestamp": "02/Dec/2014:12:18:29 +0900", "verb": "GET", "request": "/manager/html", … "agent": "\"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/
  52. ઃఆɿfilter !52 filter { grok { match => { "message"

    => "%{COMBINEDAPACHELOG}" } break_on_match => false } date { match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"] locale => en } geoip { source => ["clientip"] } useragent { source => "agent" target => "useragent" } }
  53. ೔෇ͷύʔε 53 {… "@timestamp": "2015-04-10T09:07:49.325Z", … "timestamp": "02/Dec/2014:12:18:29 +0900", …

    } {… "@timestamp": "2014-12-02T03:18:29.000Z", … "timestamp": "02/Dec/2014:12:18:29 +0900", … }
  54. ઃఆɿfilter !54 filter { grok { match => { "message"

    => "%{COMBINEDAPACHELOG}" } break_on_match => false } date { match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"] locale => en } geoip { source => ["clientip"] } useragent { source => "agent" target => "useragent" } }
  55. IP͔ΒҢ౓ܦ౓ͳͲ෇༩ 55 "clientip": "189.120.xx.xx", "clientip": "189.120.xx.xx", "geoip": { "ip": “189.120.xxx.xxx”,

    … "country_name": "Brazil", "continent_code": "SA", "region_name": "27", "city_name": "São Paulo", "latitude":
  56. ઃఆɿfilter !56 filter { grok { match => { "message"

    => "%{COMBINEDAPACHELOG}" } break_on_match => false } date { match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"] locale => en } geoip { source => ["clientip"] } useragent { source => "agent" target => "useragent" } }
  57. ϢʔβΤʔδΣϯτͷύʔε 57 "agent": "\"Mozilla/5.0 (Windows NT 5.1; rv: 5.0) Gecko/20100101

    Firefox/5.0\"" "agent": "\"Mozilla/5.0 (Windows NT 5.1; rv: 5.0) Gecko/20100101 Firefox/5.0\"" "useragent": { "name": "Firefox", "os": "Windows XP", "os_name": "Windows XP", "device": "Other", "major": "5", "minor": "0"
  58. ઃఆɿoutput 58 output { elasticsearch { hosts => ["localhost"] index

    => “demo_access_log-%{+YYYY.MM.dd}” } }
  59. ࢀߟจݙ • Elasticsearch - The Definitive guide ‒ http://www.elastic.co/guide/en/elasticsearch/guide/current/index.html •

    ॻ੶ʢ೔ຊޠʣ ‒ σʔλ෼ੳج൫ߏஙೖ໳ ‒ Elasticsearch࣮ફΨΠυ !59
  60. ࢀߟαΠτ • Ϣʔεέʔε • https://www.elastic.co/use-cases • DiscussʢWebϑΥʔϥϜʣ • https://discuss.elastic.co •

    Elastic{ON}ͷϏσΦͱࢿྉ • https://www.elastic.co/elasticon/videos • αϙʔτϝχϡʔ • https://www.elastic.co/subscriptions !60
  61. Thanks for listening! Q & A We’re hiring! https://www.elastic.co/about/careers/ We’re

    helping! https://www.elastic.co/subscriptions http://training.elastic.co
  62. QAͰ঺հͨ͠ػೳ

  63. !63 Filebeatͷೖྗ͕ଟ༷ʹ • TCP Input ‒ SSL/TLSΛαϙʔτ • UDP Input

    • Syslog Input ‒ BSD RFC3164Λαϙʔτ ‒ ϓϩτίϧ͸TCPͱUDP filebeat.inputs:
 - type: tcp
 max_message_size: 10MiB
 host: "localhost:9000" filebeat.inputs:
 - type: udp
 max_message_size: 10KiB
 host: "localhost:8080" filebeat.inputs:
 - type: syslog
 protocol.tcp:
 host: "localhost:9000" filebeat.yml
  64. !64 Data Rollups • API for creating an Elasticsearch process

    to periodically store aggregate statistics • Primary benefit is space savings ◦ Faster queries ◦ Potentially less nodes to manage ◦ Smaller snapshots ◦ Longer retention times ◦ etc. • Query rolled up data and “live” data together in a single query. Rollups API (6.3 - Experimental) • ఆظతʹ౷ܭσʔλΛू໿ͯ͠อଘ͢ΔElasticsearchͷJobΛొ࿥ • ओͳར఺͸༰ྔͷ࡟ݮ • σʔλ͕গͳ͘ͳΔͨΊ ◦ Query͕ΑΓߴ଎ʹ ◦ গͳ͍ϊʔυͰσʔλΛ؅ཧ ◦ Snapshot͕ΑΓখ͘͞ ◦ σʔλͷอ࣋ظ͕ؒΑΓ௕͘ • 1ͭͷΫΤϦͰϩʔϧΞοϓͨ͠σʔλͱͯ͠ͳ͍σʔλΛ໰͍߹Θͤ ༰ྔ͕ɻɻɻ X-Pack feature (Basic, free)
  65. !65 Raw Minute Hour Day Docs: 9,041,000 1,448,285 49,554 8,447

    Size: 2.23gb 1.25gb 48.40mb 9.10mb Docs % : -83.98% -99.45% -99.91% Size %: -43.68% -97.84% -99.59% (avg ~200 docs per minute, 32 days of data, single host) (20 grouping fields, 62 numerics @ min/max/avg == 186 metrics) Rolling up Metricbeat data ༰ྔ࡟ݮͷҰྫ MetricbeatͷϩʔϧΞοϓ (ฏۉ ~200 docs/෼ɺ32೔ؒɺ1αʔόʔ) (20ݸͷάϧʔϓϑΟʔϧυɺ62ݸͷ਺஋ @ min/max/avg == 186 metrics) X-Pack feature (Basic, free)