Elastic Stackの紹介

ヒカラボさんの大阪イベントで利用したスライドです。

Transcript

1. ‹#› 2016/11/15 Evangelist at Elastic Jun Ohtani @johtani Elastic Stackͷ঺հ
2. ‹#›
3. ΞδΣϯμ • Elastic stackͷ঺հ • ֓ཁ • Ϣʔεέʔε • Elastic stackͷଓ͚ํ 3
4. about • Me, Jun Ohtani / Technical Advocate ‒ lucene-gosenίϛολʔ ‒ ElasticSearch Server೔ຊޠ൛ͷ຋༁ ‒ http://blog.johtani.info
 • Elasticsearch, founded in 2012 ‒ Products: Elasticsearch, Logstash, Kibana, Beats 
 X-Pack(Marvel, Shield, Watcher, Graph)
 Professional services: Support & development subscriptions ‒ Trainings 4
5. ࣭໰ͦͷ1 • ElasticsearchΛ஌͍ͬͯΔ͔Ͳ͏͔ • ฉ͍ͨ͜ͱ͢Βͳ͍ • ฉ͍ͨ͜ͱ͸͋Δ͕ɺ৮ͬͨ͜ͱ͸ͳ͍ • ฉ͍ͨ͜ͱ͸͋Δ͠ɺগ͚ͩ͠ࢼͨ͠ • ຊ൪؀ڥͰӡ༻த • ฉ͍ͨ͜ͱʁͦΕͲ͜Ζ͔ɺύονʢϓϧϦΫʣॻ͖·ͬͯ͘Δ 5
6. ࣭໰ͦͷ2 • ElasticsearchҎ֎ͷElastic StackΛ࢖ͬͨ͜ͱ͕͋Δ • ࢖ͬͨ͜ͱͳ͍ • Logstash • BeatsγϦʔζ • Kibana • X-Pack 6
7. ࣭໰ͦͷ3 • ElasticsearchҎ֎ͷElastic Stackʹ·ͭΘΔԿ͔Λ։ൃͨ͜͠ͱ͕͋Δ • Logstash • BeatsγϦʔζ • Kibana • Elasticsearch • X-Pack 7
8. 8 Elastic Stack
9. 5.0 is here. All new versions. All aligned.
10. 10 Kibana Elasticsearch Beats Logstash Security Alerting Monitoring Reporting X-Pack Graph
11. 11 Kibana Elasticsearch Beats Logstash X-Pack Elastic Cloud Security Alerting Monitoring Reporting Graph
12. ‹#› Ϣʔεέʔε1 ɹݕࡧͱͯ͠ͷElasticsearch
13. ϑϦʔϫʔυݕࡧ 13
14. ߜΓࠐΈ 14
15. ϋΠϥΠτ 15
16. ιʔτ 16
17. ϖʔδϯά 17
18. ूܭ 18
19. αδΣετ 19
20. elasticsearch ͞·͟·ͳܗࣜͷσʔλͰ GeoݕࡧՄೳ
 
 Ң౓ܦ౓ɺGeoHashɺ GeoShape… GEO
21. 21 Elasticsearch is the backbone across all of Wikimedia’s sites, powering billions of real-time user prefix and full-text searches every day. “ ” Chad Horohoe Software Engineering
22. ‹#› ؆୯ͳCRUD
23. σʔλొ࿥ 23 curl -XPUT localhost:9200/books/book/1 -d ' { "title" : "Elasticsearch - The definitive guide", "authors" : "Clinton Gormley", "started" : "2013-02-04", "pages" : 230 }'
24. σʔλߋ৽ 24 curl -XPUT localhost:9200/books/book/1 -d ' { "title" : "Elasticsearch - The definitive guide", "authors" : [ "Clinton Gormley", "Zachary Tong" ], "started" : "2013-02-04", "pages" : 230 }'
25. σʔλ࡟আ 25 curl -X DELETE localhost:9200/books/book/1 σʔλͷऔಘ curl —X GET localhost:9200/books/book/1 curl —X GET localhost:9200/books/book/1/_source
26. ݕࡧ 26 curl -XGET localhost:9200/books/_search?q=elasticsearch { "took" : 2, "timed_out" : false, "_shards" : { "total" : 5, "successful" : 5, "failed" : 0 }, "hits" : { "total" : 1, "max_score" : 0.076713204, "hits" : [ { "_index" : “books", "_type" : “book", "_id" : "1", "_score" : 0.076713204, "_source" : { "title" : "Elasticsearch - The definitive guide", "authors" : [ "Clinton Gormley", "Zachary Tong" ], "started" : “2013-02-04", "pages" : 230 } } ]
27. ݕࡧ - Query DSL 27 curl -XGET ‘localhost:9200/books/book/_search' -d '{ "query": { "filtered" : { "query" : { "match": { "text" : { "query" : “To Be Or Not To Be", "cutoff_frequency" : 0.01 } } }, "filter" : { "range": { "price": { "gte": 20.0 "lte": 50.0
28. ‹#› ෼ࢄߏ੒ εέʔϧ
29. Basic terms • ΠϯσοΫε ‒ σʔλͷ࿦ཧతͳू߹ɻ
 RDBͷσʔλϕʔεͷΑ͏ͳ΋ͷLogical • ϨϓϦέʔγϣϯ • ಡΈࠐΈͷεέʔϥϏϦςΟ޲্ • SPOFͷղফ • γϟʔσΟϯά • ෳ਺Ϛγϯ΁σʔλΛ෼ׂ
 ॻ͖ࠐΈͷεέʔϥϏϦςΟ޲্
 σʔλϑϩʔ੍ޚ 29
30. γϟʔυͱϨϓϦΧ 30 node 1 orders products 1 4 1 2 2 3 curl -X PUT localhost:9200/orders -d '{ "settings.index.number_of_shards" : 4 "settings.index.number_of_replicas" : 1 }' curl -X PUT localhost:9200/products -d '{ "settings.index.number_of_shards" : 2 "settings.index.number_of_replicas" : 0 }'
31. γϟʔυͱϨϓϦΧ 31 node 1 orders products 1 4 1 node 2 orders products 2 2 3 4 1 2 3
32. ࣗಈతͳ෼ࢄ 32 node 1 orders products 2 1 4 1 node 2 orders products 2 2 node 3 orders products 3 4 1 3
33. ‹#› ΫϥΠΞϯτϥΠϒϥϦ
34. ެࣜΫϥΠΞϯτϥΠϒϥϦ • Java • Ruby • PHP • Perl • Python • .NET • JavaScript • Groovy 34
35. ΫϥΠΞϯτϥΠϒϥϦʢίϛϡχςΟʣ • Clojure • Cold Fusion • Erlang • Go • Groovy • Haskell • Java • JavaScript • kotlin 35 • .NET • OCaml • Perl • PHP • Python • R • Ruby • Scala • Smalltalk • Vert.x
36. Elasticsearch - The Definitive guide
 
 http://www.elastic.co/guide/en/ elasticsearch/guide/current/index.html 36 ৄ͘͠஌Γ͍ͨํ͸
37. ‹#› Ϣʔεέʔε2 ɹղੳͱͯ͠ͷElasticsearch
38. Elastic stackʹΑΔσʔλ෼ੳ 38 σʔλ Import/Parse
 /Export Store/Search Visualize
39. 39 収集、リッチ化、転送 ログおよび数値指標データ センサーおよびデバイスデータ Webおよびソーシャルデータ データストアおよびストリーム 分析 Elasticsearch + 任意のデータストア アラート Watcher+任意の通知ツール 監視 Marvel+任意の監視ツール アーカイブ Hadoop+任意のクラウドストレージプラットフ ォーム
40. Logstash in 10 seconds • ϩάɾσʔλͷऩूɾ؅ཧ • ऩूɺύʔεɾՃ޻ɺૹग़ • ΦʔϓϯιʔεɿApache License 2.0 • Ruby app (JRuby) 40
41. Logstash architecture 41 Input Output Filter ? ? collect and split alter and enrich store and visualize
42. ઃఆ 42 input { … } filter { … } output { … }
43. ઃఆɿinput 43 input { file { path => “/Users/johtani/sample/*_log" start_position => "beginning" } }
44. 1ߦ1σʔλ 189.120.xx.xx - - [02/Dec/2014:12:18:29 +0900] "GET /manager/html HTTP/ 1.1" 404 274 "-" "Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0" 44
45. ઃఆɿfilter 45 filter { grok { match => { "message" => "%{COMBINEDAPACHELOG}" } break_on_match => false } date { match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"] locale => en } geoip { source => ["clientip"] } useragent { source => "agent" target => "useragent" } }
46. ύʔε 46 189.120.xx.xx - - [02/Dec/2014:12:18:29 +0900] "GET /manager/html HTTP/1.1" 404 274 "-" "Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0" {… "@timestamp": "2015-04-10T09:07:49.325Z", "clientip": "189.120.xx.xx", "ident": "-", "auth": "-", "timestamp": "02/Dec/2014:12:18:29 +0900", "verb": "GET", "request": "/manager/html", … "agent": "\"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/
47. ઃఆɿfilter 47 filter { grok { match => { "message" => "%{COMBINEDAPACHELOG}" } break_on_match => false } date { match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"] locale => en } geoip { source => ["clientip"] } useragent { source => "agent" target => "useragent" } }
48. ೔෇ͷύʔε 48 {… "@timestamp": "2015-04-10T09:07:49.325Z", … "timestamp": "02/Dec/2014:12:18:29 +0900", … } {… "@timestamp": "2014-12-02T03:18:29.000Z", … "timestamp": "02/Dec/2014:12:18:29 +0900", … }
49. ઃఆɿfilter 49 filter { grok { match => { "message" => "%{COMBINEDAPACHELOG}" } break_on_match => false } date { match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"] locale => en } geoip { source => ["clientip"] } useragent { source => "agent" target => "useragent" } }
50. IP͔ΒҢ౓ܦ౓ͳͲ෇༩ 50 "clientip": "189.120.xx.xx", "clientip": "189.120.xx.xx", "geoip": { "ip": “189.120.xxx.xxx”, … "country_name": "Brazil", "continent_code": "SA", "region_name": "27", "city_name": "São Paulo", "latitude":
51. ઃఆɿfilter 51 filter { grok { match => { "message" => "%{COMBINEDAPACHELOG}" } break_on_match => false } date { match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"] locale => en } geoip { source => ["clientip"] } useragent { source => "agent" target => "useragent" } }
52. ϢʔβΤʔδΣϯτͷύʔε 52 "agent": "\"Mozilla/5.0 (Windows NT 5.1; rv: 5.0) Gecko/20100101 Firefox/5.0\"" "agent": "\"Mozilla/5.0 (Windows NT 5.1; rv: 5.0) Gecko/20100101 Firefox/5.0\"" "useragent": { "name": "Firefox", "os": "Windows XP", "os_name": "Windows XP", "device": "Other", "major": "5", "minor": "0"
53. ઃఆɿoutput 53 output { elasticsearch { hosts => ["localhost"] index => “demo_access_log-%{+YYYY.MM.dd}” } }
54. KibanaͰՄࢹԽ 54
55. 55 Elasticsearch, Logstash, and Kibana allow for real-time indexing, search, and analytics for over 300 million events per day. This protects our network, services, and systems from security threats. “ ” Jeff Bryner, Security Engineer
56. beats
57. Capture the Packet Packetbeat
58. Capture the Packet Packetbeat
59. Welcome to 1998 winlogbeat
60. Now winlogbeat
61. elasticsearch-hadoop 61 - •  D E H •  PD ecd ER •  g D •  CH •  Ca M DMS D FERC
63. 63 Elastic stackͷ ଓ͚ํ
64. ‹#› ৘ใऩू
65. ެࣜͷࢀߟαΠτ • Ϣʔεέʔε • https://www.elastic.co/use-cases • DiscussʢWebϑΥʔϥϜʣ • https://discuss.elastic.co • Elastic{ON}ͷϏσΦͱࢿྉ • https://www.elastic.co/elasticon/videos • αϙʔτϝχϡʔ • https://www.elastic.co/subscriptions 65
66. ࢀߟจݙ • Elasticsearch - The Definitive guide ‒ http://www.elastic.co/guide/en/elasticsearch/guide/current/index.html • ॻ੶ʢ೔ຊޠʣ ‒ ElasticSearchServer೔ຊޠ൛
 αʔό/ΠϯϑϥΤϯδχΞ
 ɹཆ੒ಡຊɹϩάऩू 66
67. meetup.comʢษڧձʣ 67
68. ͍ΖΜͳϒϩά • $shibayu36->blog; • http://blog.shibayu36.org/archive/category/elasticsearch • Wantedly Engineer Blog • https://www.wantedly.com/companies/wantedly/post_articles/30216 • Hello Elasticsearch! • https://medium.com/hello-elasticsearch • ෆՄࢹ఺ • http://code46.hatenablog.com/entry/2014/01/21/115620 68
69. ΧϯϑΝϨϯε • ΦʔϓϯιʔεΧϯϑΝϨϯε
 2017 Osaka • ೔࣌ɿ2017/01/27-28 • ৔ॴɿେࡕ࢈ۀ૑଄ؗ • ηογϣϯͱϒʔε • http://www.ospn.jp/osc2017- osaka/ 69
70. ‹#› ৘ใڞ༗
71. ίϛϡχςΟ׆ಈ • ษڧձʢͰൃදʣ • ษڧձʢͰ࠙਌ձʹ΋ࢀՃʣ • ϑΥʔϥϜʹࢀՃʢͯ͠ճ౴ʣ • Issue/Pull Requestͷ࡞੒ • υΩϡϝϯτͷमਖ਼ͱ͔΋͋Γ·͢Αʂ 71
72. Thanks for listening! Q & A 72 We’re hiring! https://www.elastic.co/about/careers/ We’re helping! https://www.elastic.co/subscriptions http://training.elastic.co