Slide 1
Slide 1 text
‹#›
2016/11/15
Evangelist at Elastic
Jun Ohtani @johtani
Elastic Stackͷհ
Slide 2
Slide 2 text
‹#›
Slide 3
Slide 3 text
ΞδΣϯμ
• Elastic stackͷհ
• ֓ཁ
• Ϣʔεέʔε
• Elastic stackͷଓ͚ํ
3
Slide 4
Slide 4 text
about
• Me, Jun Ohtani / Technical Advocate
‒ lucene-gosenίϛολʔ
‒ ElasticSearch Serverຊޠ൛ͷ༁
‒ http://blog.johtani.info
• Elasticsearch, founded in 2012
‒ Products: Elasticsearch, Logstash, Kibana, Beats
X-Pack(Marvel, Shield, Watcher, Graph)
Professional services: Support & development subscriptions
‒ Trainings
4
Slide 5
Slide 5 text
࣭ͦͷ1
• ElasticsearchΛ͍ͬͯΔ͔Ͳ͏͔
• ฉ͍ͨ͜ͱ͢Βͳ͍
• ฉ͍ͨ͜ͱ͋Δ͕ɺ৮ͬͨ͜ͱͳ͍
• ฉ͍ͨ͜ͱ͋Δ͠ɺগ͚ͩ͠ࢼͨ͠
• ຊ൪ڥͰӡ༻த
• ฉ͍ͨ͜ͱʁͦΕͲ͜Ζ͔ɺύονʢϓϧϦΫʣॻ͖·ͬͯ͘Δ
5
Slide 6
Slide 6 text
࣭ͦͷ2
• ElasticsearchҎ֎ͷElastic StackΛͬͨ͜ͱ͕͋Δ
• ͬͨ͜ͱͳ͍
• Logstash
• BeatsγϦʔζ
• Kibana
• X-Pack
6
Slide 7
Slide 7 text
࣭ͦͷ3
• ElasticsearchҎ֎ͷElastic Stackʹ·ͭΘΔԿ͔Λ։ൃͨ͜͠ͱ͕͋Δ
• Logstash
• BeatsγϦʔζ
• Kibana
• Elasticsearch
• X-Pack
7
Slide 8
Slide 8 text
8
Elastic Stack
Slide 9
Slide 9 text
5.0 is here.
All new versions.
All aligned.
Slide 10
Slide 10 text
10
Kibana
Elasticsearch
Beats Logstash
Security
Alerting
Monitoring
Reporting
X-Pack
Graph
Slide 11
Slide 11 text
11
Kibana
Elasticsearch
Beats Logstash
X-Pack Elastic Cloud
Security
Alerting
Monitoring
Reporting
Graph
Slide 12
Slide 12 text
‹#›
Ϣʔεέʔε1
ɹݕࡧͱͯ͠ͷElasticsearch
Slide 13
Slide 13 text
ϑϦʔϫʔυݕࡧ
13
Slide 14
Slide 14 text
ߜΓࠐΈ
14
Slide 15
Slide 15 text
ϋΠϥΠτ
15
Slide 16
Slide 16 text
ιʔτ
16
Slide 17
Slide 17 text
ϖʔδϯά
17
Slide 18
Slide 18 text
ूܭ
18
Slide 19
Slide 19 text
αδΣετ
19
Slide 20
Slide 20 text
elasticsearch
͞·͟·ͳܗࣜͷσʔλͰ
GeoݕࡧՄೳ
ҢܦɺGeoHashɺ
GeoShape…
GEO
Slide 21
Slide 21 text
21
Elasticsearch is the
backbone across all of
Wikimedia’s sites, powering
billions of real-time user
prefix and full-text searches
every day.
“
”
Chad Horohoe
Software Engineering
Slide 22
Slide 22 text
‹#›
؆୯ͳCRUD
Slide 23
Slide 23 text
σʔλొ
23
curl -XPUT localhost:9200/books/book/1 -d '
{
"title" : "Elasticsearch - The definitive guide",
"authors" : "Clinton Gormley",
"started" : "2013-02-04",
"pages" : 230
}'
Slide 24
Slide 24 text
σʔλߋ৽
24
curl -XPUT localhost:9200/books/book/1 -d '
{
"title" : "Elasticsearch - The definitive guide",
"authors" : [ "Clinton Gormley", "Zachary Tong" ],
"started" : "2013-02-04",
"pages" : 230
}'
Slide 25
Slide 25 text
σʔλআ
25
curl -X DELETE localhost:9200/books/book/1
σʔλͷऔಘ
curl —X GET localhost:9200/books/book/1
curl —X GET localhost:9200/books/book/1/_source
Slide 26
Slide 26 text
ݕࡧ
26
curl -XGET localhost:9200/books/_search?q=elasticsearch
{
"took" : 2, "timed_out" : false,
"_shards" : { "total" : 5, "successful" : 5, "failed" : 0 },
"hits" : {
"total" : 1, "max_score" : 0.076713204,
"hits" : [ {
"_index" : “books", "_type" : “book", "_id" : "1",
"_score" : 0.076713204, "_source" : {
"title" : "Elasticsearch - The definitive guide",
"authors" : [ "Clinton Gormley", "Zachary Tong" ],
"started" : “2013-02-04", "pages" : 230
}
} ]
Slide 27
Slide 27 text
ݕࡧ - Query DSL
27
curl -XGET ‘localhost:9200/books/book/_search' -d '{
"query": {
"filtered" : {
"query" : {
"match": {
"text" : {
"query" : “To Be Or Not To Be",
"cutoff_frequency" : 0.01
}
}
},
"filter" : {
"range": {
"price": {
"gte": 20.0
"lte": 50.0
Slide 28
Slide 28 text
‹#›
ࢄߏ
εέʔϧ
Slide 29
Slide 29 text
Basic terms
• ΠϯσοΫε
‒ σʔλͷཧతͳू߹ɻ
RDBͷσʔλϕʔεͷΑ͏ͳͷLogical
• ϨϓϦέʔγϣϯ
• ಡΈࠐΈͷεέʔϥϏϦςΟ্
• SPOFͷղফ
• γϟʔσΟϯά
• ෳϚγϯσʔλΛׂ
ॻ͖ࠐΈͷεέʔϥϏϦςΟ্
σʔλϑϩʔ੍ޚ
29
Slide 30
Slide 30 text
γϟʔυͱϨϓϦΧ
30
node 1
orders
products
1
4
1 2
2
3
curl -X PUT localhost:9200/orders -d '{
"settings.index.number_of_shards" : 4
"settings.index.number_of_replicas" : 1
}'
curl -X PUT localhost:9200/products -d '{
"settings.index.number_of_shards" : 2
"settings.index.number_of_replicas" : 0
}'
Slide 31
Slide 31 text
γϟʔυͱϨϓϦΧ
31
node 1
orders
products
1
4
1
node 2
orders
products
2
2
3 4
1 2
3
Slide 32
Slide 32 text
ࣗಈతͳࢄ
32
node 1
orders
products
2
1
4
1
node 2
orders
products
2
2
node 3
orders
products
3 4
1
3
Slide 33
Slide 33 text
‹#›
ΫϥΠΞϯτϥΠϒϥϦ
Slide 34
Slide 34 text
ެࣜΫϥΠΞϯτϥΠϒϥϦ
• Java
• Ruby
• PHP
• Perl
• Python
• .NET
• JavaScript
• Groovy
34
Slide 35
Slide 35 text
ΫϥΠΞϯτϥΠϒϥϦʢίϛϡχςΟʣ
• Clojure
• Cold Fusion
• Erlang
• Go
• Groovy
• Haskell
• Java
• JavaScript
• kotlin
35
• .NET
• OCaml
• Perl
• PHP
• Python
• R
• Ruby
• Scala
• Smalltalk
• Vert.x
Slide 36
Slide 36 text
Elasticsearch - The Definitive guide
http://www.elastic.co/guide/en/
elasticsearch/guide/current/index.html
36
ৄ͘͠Γ͍ͨํ
Slide 37
Slide 37 text
‹#›
Ϣʔεέʔε2
ɹղੳͱͯ͠ͷElasticsearch
Slide 38
Slide 38 text
Elastic stackʹΑΔσʔλੳ
38
σʔλ Import/Parse
/Export
Store/Search Visualize
Slide 39
Slide 39 text
39
収集、リッチ化、転送
ログおよび数値指標データ
センサーおよびデバイスデータ
Webおよびソーシャルデータ
データストアおよびストリーム
分析
Elasticsearch + 任意のデータストア
アラート
Watcher+任意の通知ツール
監視
Marvel+任意の監視ツール
アーカイブ
Hadoop+任意のクラウドストレージプラットフ
ォーム
Slide 40
Slide 40 text
Logstash in 10 seconds
• ϩάɾσʔλͷऩूɾཧ
• ऩूɺύʔεɾՃɺૹग़
• ΦʔϓϯιʔεɿApache License 2.0
• Ruby app (JRuby)
40
Slide 41
Slide 41 text
Logstash architecture
41
Input Output
Filter
? ?
collect and split alter and enrich store and visualize
Slide 42
Slide 42 text
ઃఆ
42
input {
…
}
filter {
…
}
output {
…
}
Slide 43
Slide 43 text
ઃఆɿinput
43
input {
file {
path => “/Users/johtani/sample/*_log"
start_position => "beginning"
}
}
Slide 44
Slide 44 text
1ߦ1σʔλ
189.120.xx.xx - - [02/Dec/2014:12:18:29 +0900] "GET /manager/html HTTP/
1.1" 404 274 "-" "Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101
Firefox/5.0"
44
Slide 45
Slide 45 text
ઃఆɿfilter
45
filter {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
break_on_match => false
}
date {
match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"]
locale => en
}
geoip { source => ["clientip"] }
useragent {
source => "agent"
target => "useragent"
}
}
Slide 46
Slide 46 text
ύʔε
46
189.120.xx.xx - - [02/Dec/2014:12:18:29 +0900] "GET /manager/html HTTP/1.1"
404 274 "-" "Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0"
{…
"@timestamp": "2015-04-10T09:07:49.325Z",
"clientip": "189.120.xx.xx",
"ident": "-",
"auth": "-",
"timestamp": "02/Dec/2014:12:18:29 +0900",
"verb": "GET",
"request": "/manager/html",
…
"agent": "\"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/
Slide 47
Slide 47 text
ઃఆɿfilter
47
filter {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
break_on_match => false
}
date {
match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"]
locale => en
}
geoip { source => ["clientip"] }
useragent {
source => "agent"
target => "useragent"
}
}
Slide 48
Slide 48 text
ͷύʔε
48
{…
"@timestamp": "2015-04-10T09:07:49.325Z",
…
"timestamp": "02/Dec/2014:12:18:29 +0900",
…
}
{…
"@timestamp": "2014-12-02T03:18:29.000Z",
…
"timestamp": "02/Dec/2014:12:18:29 +0900",
…
}
Slide 49
Slide 49 text
ઃఆɿfilter
49
filter {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
break_on_match => false
}
date {
match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"]
locale => en
}
geoip { source => ["clientip"] }
useragent {
source => "agent"
target => "useragent"
}
}
Slide 50
Slide 50 text
IP͔ΒҢܦͳͲ༩
50
"clientip": "189.120.xx.xx",
"clientip": "189.120.xx.xx",
"geoip": {
"ip": “189.120.xxx.xxx”,
…
"country_name": "Brazil",
"continent_code": "SA",
"region_name": "27",
"city_name": "São Paulo",
"latitude":
Slide 51
Slide 51 text
ઃఆɿfilter
51
filter {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
break_on_match => false
}
date {
match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"]
locale => en
}
geoip { source => ["clientip"] }
useragent {
source => "agent"
target => "useragent"
}
}
Slide 52
Slide 52 text
ϢʔβΤʔδΣϯτͷύʔε
52
"agent": "\"Mozilla/5.0 (Windows NT 5.1; rv:
5.0) Gecko/20100101 Firefox/5.0\""
"agent": "\"Mozilla/5.0 (Windows NT 5.1; rv:
5.0) Gecko/20100101 Firefox/5.0\""
"useragent": {
"name": "Firefox",
"os": "Windows XP",
"os_name": "Windows XP",
"device": "Other",
"major": "5",
"minor": "0"
Slide 53
Slide 53 text
ઃఆɿoutput
53
output {
elasticsearch {
hosts => ["localhost"]
index => “demo_access_log-%{+YYYY.MM.dd}”
}
}
Slide 54
Slide 54 text
KibanaͰՄࢹԽ
54
Slide 55
Slide 55 text
55
Elasticsearch, Logstash, and
Kibana allow for real-time
indexing, search, and
analytics for over 300 million
events per day. This protects
our network, services, and
systems from security
threats.
“
”
Jeff Bryner, Security Engineer
Slide 56
Slide 56 text
beats
Slide 57
Slide 57 text
Capture the
Packet
Packetbeat
Slide 58
Slide 58 text
Capture the
Packet
Packetbeat
Slide 59
Slide 59 text
Welcome
to 1998
winlogbeat
Slide 60
Slide 60 text
Now
winlogbeat
Slide 61
Slide 61 text
elasticsearch-hadoop
61
-
• D E H
• PD ecd
ER
• g D
•
CH
• Ca M DMS
D FERC
Slide 62
Slide 62 text
No content
Slide 63
Slide 63 text
63
Elastic stackͷ
ଓ͚ํ
Slide 64
Slide 64 text
‹#›
ใऩू
Slide 65
Slide 65 text
ެࣜͷࢀߟαΠτ
• Ϣʔεέʔε
• https://www.elastic.co/use-cases
• DiscussʢWebϑΥʔϥϜʣ
• https://discuss.elastic.co
• Elastic{ON}ͷϏσΦͱࢿྉ
• https://www.elastic.co/elasticon/videos
• αϙʔτϝχϡʔ
• https://www.elastic.co/subscriptions
65
Slide 66
Slide 66 text
ࢀߟจݙ
• Elasticsearch - The Definitive guide
‒ http://www.elastic.co/guide/en/elasticsearch/guide/current/index.html
• ॻ੶ʢຊޠʣ
‒ ElasticSearchServerຊޠ൛
αʔό/ΠϯϑϥΤϯδχΞ
ɹཆಡຊɹϩάऩू
66
Slide 67
Slide 67 text
meetup.comʢษڧձʣ
67
Slide 68
Slide 68 text
͍ΖΜͳϒϩά
• $shibayu36->blog;
• http://blog.shibayu36.org/archive/category/elasticsearch
• Wantedly Engineer Blog
• https://www.wantedly.com/companies/wantedly/post_articles/30216
• Hello Elasticsearch!
• https://medium.com/hello-elasticsearch
• ෆՄࢹ
• http://code46.hatenablog.com/entry/2014/01/21/115620
68
Slide 69
Slide 69 text
ΧϯϑΝϨϯε
• ΦʔϓϯιʔεΧϯϑΝϨϯε
2017 Osaka
• ࣌ɿ2017/01/27-28
• ॴɿେࡕ࢈ۀؗ
• ηογϣϯͱϒʔε
• http://www.ospn.jp/osc2017-
osaka/
69
Slide 70
Slide 70 text
‹#›
ใڞ༗
Slide 71
Slide 71 text
ίϛϡχςΟ׆ಈ
• ษڧձʢͰൃදʣ
• ษڧձʢͰ࠙ձʹࢀՃʣ
• ϑΥʔϥϜʹࢀՃʢͯ͠ճʣ
• Issue/Pull Requestͷ࡞
• υΩϡϝϯτͷमਖ਼ͱ͔͋Γ·͢Αʂ
71
Slide 72
Slide 72 text
Thanks for listening!
Q & A
72
We’re hiring!
https://www.elastic.co/about/careers/
We’re helping!
https://www.elastic.co/subscriptions
http://training.elastic.co