Intro Elastic Stack at Telemetry WG

2018/06/29にTelemetryWGで使用したスライド。

Transcript

1. !1 Jun Ohtani 2018/06/29 at TelemetryWG @johtani Elastic stackͷ঺հ
2. ‹#›
3. about • Me, Jun Ohtani / Technical Advocate ‒ lucene-gosenίϛολʔ ‒ ElasticSearch Server೔ຊޠ൛ͷ຋༁ ‒ http://blog.johtani.info
 • Elasticsearch, founded in 2012 ‒ Products: Elasticsearch, Logstash, Kibana, Beats 
 X-Pack, Elastic Cloud
 Professional services: Support & development subscriptions ‒ Trainings, Consulting, SaaS !3
4. 4 ElasticελοΫ
5. !5 Elastic Stack 
   Φʔϓϯιʔε
   ΛϦϦʔε
6. Πϯετʔϧ͸؆୯
   &MBTUJD
   4UBDLΛ֦ு
   αϒεΫϦϓγϣϯʹؚΉ X-Pack !6 Security Alerting Monitoring Reporting Graph Machine Learning
7. ϝτϦΫε/ϩάղੳΛ ࢝ΊͯΈΑ͏
8. ϝτϦΫεɾϩάͷ෼ੳʢ؆қ൛ʣ !8 σʔλ Import Parse/
 Store/Search Visualize
9. ܰྔσʔλγούʔ 9 Beats
10. 10 Beats ܰྔσʔλγούʔ ιʔε͔ΒσʔλΛసૹ సૹ͠Elasticsearchʹू໿ ม׵ͱύʔεͷͨΊ Logstashʹసૹ Elastic Cloudʹసૹ Libbeat: ΧελϜbeatsͷͨ ΊͷAPIϑϨʔϜϫʔΫ 30Ҏ্ͷίϛϡχςΟbeats
11. 11 FILEBEAT ϩάϑΝΠϧ METRICBEAT ϝτϦοΫ৘ใ PACKETBEAT ωοτϫʔΫ WINGLOGBEAT WindowΠϕϯτ ͞Βʹ30Λ௒͑ΔίϛϡχςΟ Beats͕͋Γɺ૿Ճத Apachebeat, dockbeat, httpbeat, mysqlbeat, nginxbeat, redis beats, twitterbeat, and more
12. Collect system and application metrics Metricbeat
13. lots of modules Metricbeat
14. tail log from file Filebeat
15. many modules Filebeat
16. Capture the Packet Packetbeat
17. Capture the Packet Packetbeat
18. Welcome to 1998 winlogbeat
19. Now winlogbeat
20. 20 Elasticsearch
21. ݕࡧͱͯ͠ͷ
 Elasticsearch
22. ؆୯ͳCRUD
23. σʔλొ࿥ 23 curl -XPUT localhost:9200/books/book/1 -d ' { "title" : "Elasticsearch - The definitive guide", "authors" : "Clinton Gormley", "started" : "2013-02-04", "pages" : 230 }'
24. σʔλߋ৽ 24 curl -XPUT localhost:9200/books/book/1 -d ' { "title" : "Elasticsearch - The definitive guide", "authors" : [ "Clinton Gormley", "Zachary Tong" ], "started" : "2013-02-04", "pages" : 230 }'
25. σʔλ࡟আ !25 curl -X DELETE localhost:9200/books/book/1 σʔλͷऔಘ curl —X GET localhost:9200/books/book/1 curl —X GET localhost:9200/books/book/1/_source
26. ݕࡧ !26 curl -XGET localhost:9200/books/_search?q=elasticsearch { "took" : 2, "timed_out" : false, "_shards" : { "total" : 5, "successful" : 5, "failed" : 0 }, "hits" : { "total" : 1, "max_score" : 0.076713204, "hits" : [ { "_index" : “books", "_type" : “book", "_id" : "1", "_score" : 0.076713204, "_source" : { "title" : "Elasticsearch - The definitive guide", "authors" : [ "Clinton Gormley", "Zachary Tong" ], "started" : “2013-02-04", "pages" : 230 } } ]
27. ݕࡧ - Query DSL !27 curl -XGET ‘localhost:9200/books/book/_search' -d '{ "query": { "filtered" : { "query" : { "match": { "text" : { "query" : “To Be Or Not To Be", "cutoff_frequency" : 0.01 } } }, "filter" : { "range": { "price": { "gte": 20.0 "lte": 50.0
28. ෼ࢄߏ੒ɺ
 εέʔϧ
29. Basic terms • ΠϯσοΫε ‒ σʔλͷ࿦ཧతͳू߹ɻ
 RDBͷσʔλϕʔεͷΑ͏ͳ΋ͷLogical • ϨϓϦέʔγϣϯ • ಡΈࠐΈͷεέʔϥϏϦςΟ޲্ • SPOFͷղফ • γϟʔσΟϯά • ෳ਺Ϛγϯ΁σʔλΛ෼ׂ
 ॻ͖ࠐΈͷεέʔϥϏϦςΟ޲্
 σʔλϑϩʔ੍ޚ !29
30. γϟʔυͱϨϓϦΧ !30 node 1 orders products 1 4 1 2 2 3 curl -X PUT localhost:9200/orders -d '{ "settings.index.number_of_shards" : 4 "settings.index.number_of_replicas" : 1 }' curl -X PUT localhost:9200/products -d '{ "settings.index.number_of_shards" : 2 "settings.index.number_of_replicas" : 0 }'
31. γϟʔυͱϨϓϦΧ !31 node 1 orders products 1 4 1 node 2 orders products 2 2 3 4 1 2 3
32. ࣗಈతͳ෼ࢄ !32 node 1 orders products 2 1 4 1 node 2 orders products 2 2 node 3 orders products 3 4 1 3
33. ͦͷଞͷػೳ
34. elasticsearch ͞·͟·ͳܗࣜͷσʔλͰ GeoݕࡧՄೳ
 
 Ң౓ܦ౓ɺGeoHashɺ GeoShape… GEO
35. Ecosystem • Plugins ‒ ϓϥάΠϯʹΑΔػೳͷ௥Ճ • ΫϥΠΞϯτϥΠϒϥϦ • Java, Ruby, python, php, perl, javascript, .NET • Scala, clojure, go !35
36. Elasticsearch - The Definitive guide
 
 http://www.elastic.co/guide/en/ elasticsearch/guide/current/index.html 36 ৄ͘͠஌Γ͍ͨํ͸
37. 37 KibanaͰՄࢹԽ
38. Kibana 5 • ElasticsearchͷσʔλΛՄࢹԽ • Node.js server & JavaScript • Apache License 2.0 • Elastic Stackͷ૭ͷ໾ׂ • ༷ʑͳGUIΛPluginͱ͍ͯެ։ • MarvelɺSenseɺTimelionͳͲ !38
39. Kibana 5 39
40. Combining Search and Analytics !40
41. σϞ for Kibana5 Access Log 41
42. ຊ֨తʹղੳΛߦ͏ʹ͸ʁ
44. 44 Logstash
45. Logstash in 10 seconds • ϩάɾσʔλͷऩूɾ؅ཧ • ऩूɺύʔεɾՃ޻ɺૹग़ • ΦʔϓϯιʔεɿApache License 2.0 • Ruby app (JRuby) !45
46. Logstash architecture !46 Input Output Filter ? ? collect and split alter and enrich store and visualize
47. ઃఆ 47 input { … } filter { … } output { … }
48. ઃఆɿinput 48 input { file { path => “/Users/johtani/sample/*_log" start_position => "beginning" } }
49. 1ߦ1σʔλ 189.120.xx.xx - - [02/Dec/2014:12:18:29 +0900] "GET /manager/html HTTP/ 1.1" 404 274 "-" "Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0" 49
50. ઃఆɿfilter 50 filter { grok { match => { "message" => "%{COMBINEDAPACHELOG}" } break_on_match => false } date { match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"] locale => en } geoip { source => ["clientip"] } useragent { source => "agent" target => "useragent" } }
51. ύʔε !51 189.120.xx.xx - - [02/Dec/2014:12:18:29 +0900] "GET /manager/html HTTP/1.1" 404 274 "-" "Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0" {… "@timestamp": "2015-04-10T09:07:49.325Z", "clientip": "189.120.xx.xx", "ident": "-", "auth": "-", "timestamp": "02/Dec/2014:12:18:29 +0900", "verb": "GET", "request": "/manager/html", … "agent": "\"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/
52. ઃఆɿfilter !52 filter { grok { match => { "message" => "%{COMBINEDAPACHELOG}" } break_on_match => false } date { match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"] locale => en } geoip { source => ["clientip"] } useragent { source => "agent" target => "useragent" } }
53. ೔෇ͷύʔε 53 {… "@timestamp": "2015-04-10T09:07:49.325Z", … "timestamp": "02/Dec/2014:12:18:29 +0900", … } {… "@timestamp": "2014-12-02T03:18:29.000Z", … "timestamp": "02/Dec/2014:12:18:29 +0900", … }
54. ઃఆɿfilter !54 filter { grok { match => { "message" => "%{COMBINEDAPACHELOG}" } break_on_match => false } date { match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"] locale => en } geoip { source => ["clientip"] } useragent { source => "agent" target => "useragent" } }
55. IP͔ΒҢ౓ܦ౓ͳͲ෇༩ 55 "clientip": "189.120.xx.xx", "clientip": "189.120.xx.xx", "geoip": { "ip": “189.120.xxx.xxx”, … "country_name": "Brazil", "continent_code": "SA", "region_name": "27", "city_name": "São Paulo", "latitude":
56. ઃఆɿfilter !56 filter { grok { match => { "message" => "%{COMBINEDAPACHELOG}" } break_on_match => false } date { match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"] locale => en } geoip { source => ["clientip"] } useragent { source => "agent" target => "useragent" } }
57. ϢʔβΤʔδΣϯτͷύʔε 57 "agent": "\"Mozilla/5.0 (Windows NT 5.1; rv: 5.0) Gecko/20100101 Firefox/5.0\"" "agent": "\"Mozilla/5.0 (Windows NT 5.1; rv: 5.0) Gecko/20100101 Firefox/5.0\"" "useragent": { "name": "Firefox", "os": "Windows XP", "os_name": "Windows XP", "device": "Other", "major": "5", "minor": "0"
58. ઃఆɿoutput 58 output { elasticsearch { hosts => ["localhost"] index => “demo_access_log-%{+YYYY.MM.dd}” } }
59. ࢀߟจݙ • Elasticsearch - The Definitive guide ‒ http://www.elastic.co/guide/en/elasticsearch/guide/current/index.html • ॻ੶ʢ೔ຊޠʣ ‒ σʔλ෼ੳج൫ߏஙೖ໳ ‒ Elasticsearch࣮ફΨΠυ !59
60. ࢀߟαΠτ • Ϣʔεέʔε • https://www.elastic.co/use-cases • DiscussʢWebϑΥʔϥϜʣ • https://discuss.elastic.co • Elastic{ON}ͷϏσΦͱࢿྉ • https://www.elastic.co/elasticon/videos • αϙʔτϝχϡʔ • https://www.elastic.co/subscriptions !60
61. Thanks for listening! Q & A We’re hiring! https://www.elastic.co/about/careers/ We’re helping! https://www.elastic.co/subscriptions http://training.elastic.co
62. QAͰ঺հͨ͠ػೳ
63. !63 Filebeatͷೖྗ͕ଟ༷ʹ • TCP Input ‒ SSL/TLSΛαϙʔτ • UDP Input • Syslog Input ‒ BSD RFC3164Λαϙʔτ ‒ ϓϩτίϧ͸TCPͱUDP filebeat.inputs:
 - type: tcp
 max_message_size: 10MiB
 host: "localhost:9000" filebeat.inputs:
 - type: udp
 max_message_size: 10KiB
 host: "localhost:8080" filebeat.inputs:
 - type: syslog
 protocol.tcp:
 host: "localhost:9000" filebeat.yml
64. !64 Data Rollups • API for creating an Elasticsearch process to periodically store aggregate statistics • Primary benefit is space savings ◦ Faster queries ◦ Potentially less nodes to manage ◦ Smaller snapshots ◦ Longer retention times ◦ etc. • Query rolled up data and “live” data together in a single query. Rollups API (6.3 - Experimental) • ఆظతʹ౷ܭσʔλΛू໿ͯ͠อଘ͢ΔElasticsearchͷJobΛొ࿥ • ओͳར఺͸༰ྔͷ࡟ݮ • σʔλ͕গͳ͘ͳΔͨΊ ◦ Query͕ΑΓߴ଎ʹ ◦ গͳ͍ϊʔυͰσʔλΛ؅ཧ ◦ Snapshot͕ΑΓখ͘͞ ◦ σʔλͷอ࣋ظ͕ؒΑΓ௕͘ • 1ͭͷΫΤϦͰϩʔϧΞοϓͨ͠σʔλͱͯ͠ͳ͍σʔλΛ໰͍߹Θͤ ༰ྔ͕ɻɻɻ X-Pack feature (Basic, free)
65. !65 Raw Minute Hour Day Docs: 9,041,000 1,448,285 49,554 8,447 Size: 2.23gb 1.25gb 48.40mb 9.10mb Docs % : -83.98% -99.45% -99.91% Size %: -43.68% -97.84% -99.59% (avg ~200 docs per minute, 32 days of data, single host) (20 grouping fields, 62 numerics @ min/max/avg == 186 metrics) Rolling up Metricbeat data ༰ྔ࡟ݮͷҰྫ MetricbeatͷϩʔϧΞοϓ (ฏۉ ~200 docs/෼ɺ32೔ؒɺ1αʔόʔ) (20ݸͷάϧʔϓϑΟʔϧυɺ62ݸͷ਺஋ @ min/max/avg == 186 metrics) X-Pack feature (Basic, free)