Slide 1
Slide 1 text
‹#›
2017/09/16
Evangelist at Elastic
Jun Ohtani @johtani
Elastic stackʹ͓͚Δ
ཧใͷར༻ํ๏
Slide 2
Slide 2 text
‹#›
Slide 3
Slide 3 text
ΞδΣϯμ
• Elastic Stack ͱʁ
• Elastic StackͰͷཧใͷػೳ
• LogstashͰͷGeoIP
• ElasticsearchͰͷҢܦରԠ
• KibanaͰͷਤରԠ
3
Slide 4
Slide 4 text
about
• Me, Jun Ohtani / Technical Advocate
‒ lucene-gosenίϛολʔ
‒ ElasticSearch Serverຊޠ൛ͷ༁
‒ http://blog.johtani.info
• Elasticsearch, founded in 2012
‒ Products: Elasticsearch, Logstash, Kibana, Beats
X-Pack, Elastic Cloud
Professional services: Support & development subscriptions
‒ Trainings, Consulting, SaaS
4
Slide 5
Slide 5 text
5
ElasticελοΫ
Slide 6
Slide 6 text
6
Elastic Stack
100% Φʔϓϯιʔε
ʮΤϯλʔϓϥΠζ൛ʯແ͠
όʔδϣϯ 5.0Ͱશ౷Ұ
Slide 7
Slide 7 text
7
Logstash
Slide 8
Slide 8 text
Logstash in 10 seconds
• ϩάɾσʔλͷऩूɾཧ
• ऩूɺύʔεɾՃɺૹग़
• ΦʔϓϯιʔεɿApache License 2.0
• Ruby app (JRuby)
8
Slide 9
Slide 9 text
Logstash architecture
9
Input Output
Filter
? ?
collect and split alter and enrich store and visualize
Slide 10
Slide 10 text
ઃఆɿfilter
10
filter {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
break_on_match => false
}
date {
match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"]
locale => en
}
geoip { source => ["clientip"] }
useragent {
source => "agent"
target => "useragent"
}
}
Slide 11
Slide 11 text
ύʔε
11
189.120.xx.xx - - [02/Dec/2014:12:18:29 +0900] "GET /manager/html HTTP/1.1"
404 274 "-" "Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0"
{…
"@timestamp": "2015-04-10T09:07:49.325Z",
"clientip": "189.120.xx.xx",
"ident": "-",
"auth": "-",
"timestamp": "02/Dec/2014:12:18:29 +0900",
"verb": "GET",
"request": "/manager/html",
…
"agent": "\"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/
Slide 12
Slide 12 text
12
Elasticsearch
Slide 13
Slide 13 text
ݕࡧͱͯ͠ͷ
Elasticsearch
Slide 14
Slide 14 text
Elasticsearchͱʁ
Slide 15
Slide 15 text
ϑϦʔϫʔυݕࡧ
15
Slide 16
Slide 16 text
ߜΓࠐΈ
16
Slide 17
Slide 17 text
ϋΠϥΠτ
17
Slide 18
Slide 18 text
ιʔτ
18
Slide 19
Slide 19 text
ϖʔδϯά
19
Slide 20
Slide 20 text
ूܭ
20
Slide 21
Slide 21 text
αδΣετ
21
Slide 22
Slide 22 text
Elasticsearch in 10 seconds
• εΩʔϚϑϦʔɺࢄυΩϡϝϯτετΞɺREST & JSON
• Φʔϓϯιʔε: Apache License 2.0
• ઃఆͳ͠Ͱ؆୯ʹࢼ͢͜ͱ͕Մೳ
• JavaͰ࣮ɻ֦ு༰қ
22
Slide 23
Slide 23 text
σʔλొ
23
curl -XPUT localhost:9200/books/book/1 -d '
{
"title" : "Elasticsearch - The definitive guide",
"authors" : "Clinton Gormley",
"started" : "2013-02-04",
"pages" : 230
}'
Slide 24
Slide 24 text
elasticsearch
͞·͟·ͳܗࣜͷσʔλͰ
GeoݕࡧՄೳ
ҢܦɺGeoHashɺ
GeoShape…
GEO
Slide 25
Slide 25 text
Ecosystem
• Plugins
‒ ϓϥάΠϯʹΑΔػೳͷՃ
• ΫϥΠΞϯτϥΠϒϥϦ
• Java, Ruby, python, php, perl, javascript, .NET
• Scala, clojure, go
25
Slide 26
Slide 26 text
26
KibanaͰՄࢹԽ
Slide 27
Slide 27 text
Kibana 5
• ElasticsearchͷσʔλΛՄࢹԽ
• Node.js server & JavaScript
• Apache License 2.0
• Elastic Stackͷ૭ͷׂ
• ༷ʑͳGUIΛPluginͱ͍ͯެ։
• MarvelɺSenseɺTimelionͳͲ
27
Slide 28
Slide 28 text
Kibana 5
28
Slide 29
Slide 29 text
29
X-Pack
؆୯ʹΠϯετʔϧ
Elastic StackΛ֦ு
αϒεΫϦϓγϣϯʹؚ·ΕΔ
Security
Alerting
Monitoring
Reporting
Graph
Machine Learning
Slide 30
Slide 30 text
LogstashͰͷ׆༻ํ๏
Slide 31
Slide 31 text
ઃఆɿgeoip filter
31
filter {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
break_on_match => false
}
date {
match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"]
locale => en
}
geoip { source => ["clientip"] }
useragent {
source => "agent"
target => "useragent"
}
}
Slide 32
Slide 32 text
IP͔ΒҢܦͳͲ༩
32
"clientip": "124.35.xx.xx",
"clientip": "124.35.xx.xx",
"geoip": {
"continent_name": "Asia",
"city_name": "Tokyo",
"country_iso_code": "JP",
"region_name": "Tokyo",
"location": {
"lon": 139.7559,
"lat": 35.6845
Slide 33
Slide 33 text
geoip filter
• MaxMind GeoLite2 CityσʔλϕʔεΛϥΠϒϥϦʹόϯυϧ
• IPv4ͱIPv6ͷ྆ํʹରԠ
• Autonomous System NumberͷσʔλϕʔεΓସ͑Մೳ
• MaxMindͷ༻σʔλϕʔεར༻Մೳ
• `database`ύϥϝʔλͰϑΝΠϧΛࢦఆՄೳ
33
Slide 34
Slide 34 text
ElasticsearchͰͷཧใͷ׆༻
Slide 35
Slide 35 text
Datatype
• geo_point
• ҢܦͷϙΠϯτ
• Geo·ͨڑʹج͍ͮͨσʔλͷूܭ
• ڑʹΑΔιʔτɺGeoʹج͍ͮͨؔ࿈ͷௐ
• geo_shape
• Geo ShapeσʔλʢϙϦΰϯۣܗʣ
• geohash͘͠quadtree
35
Slide 36
Slide 36 text
Sample data
Point
36
{
"location" : {
"type" : "point",
"coordinates" : [-77.03653, 38.897676]
}
}
Slide 37
Slide 37 text
Sample data
LineString
37
{
"location" : {
"type" : "linestring",
"coordinates" : [[-77.03653, 38.897676],
[-77.009051, 38.889939]]
}
}
Slide 38
Slide 38 text
Sample data
Polygon
38
{
"location" : {
"type" : "polygon",
"coordinates" : [
[ [100.0, 0.0],
[101.0, 0.0],
[101.0, 1.0],
[100.0, 1.0],
[100.0, 0.0] ]
]
}
}
Slide 39
Slide 39 text
Sample data
Geometry Collection
39
{
"location" : {
"type": "geometrycollection",
"geometries": [
{
"type": "point",
"coordinates": [100.0, 0.0]
},{
"type": "linestring",
"coordinates": [ [101.0, 0.0], [102.0, 1.0] ]
}
]
Slide 40
Slide 40 text
Queries
40
• geo_shape query
• geo_shapeͷݕࡧ
• geo_bounding_box query
• geo_pointͷݕࡧʢۣܗʹΑΔݕࡧʣ
• geo_distance/geo_distance_range query
• geo_pointͷݕࡧʢத৺ʴڑʹΑΔݕࡧʣ
• geo_polygon query
• geo_pointͷݕࡧʢϙϦΰϯʹΑΔݕࡧʣ
Slide 41
Slide 41 text
Queries
geo_shape query
41
GET /example/_search
{...
"filter": {
"geo_shape": {
"location": {
"shape": {
"type": "envelope",
"coordinates" : [[13.0, 53.0], [14.0, 52.0]]
},
"relation": "within"
}...
Slide 42
Slide 42 text
Queries
geo_bounding_box query
42
"filter" : {
"geo_bounding_box" : {
"pin.location" : {
"top_left" : {
"lat" : 40.73,
"lon" : -74.1
},
"bottom_right" : {
"lat" : 40.01,
"lon" : -71.12
}...
Slide 43
Slide 43 text
Queries
geo_distance query
43
...
"filter" : {
"geo_distance" : {
"distance" : "200km",
"pin.location" : {
"lat" : 40,
"lon" : -70
}
}
}...
Slide 44
Slide 44 text
Queries
geo_distance_range query
44
...
"filter" : {
"geo_distance_range" : {
"from" : "200km",
"to" : "400km",
"pin.location" : {
"lat" : 40,
"lon" : -70
}
}
}...
Slide 45
Slide 45 text
Queries
geo_polygon query
45
...
"filter" : {
"geo_polygon" : {
"person.location" : {
"points" : [
{"lat" : 40, "lon" : -70},
{"lat" : 30, "lon" : -80},
{"lat" : 20, "lon" : -90}
]
}
}...
Slide 46
Slide 46 text
aggregation (grouping)
• geo_bounds aggregation
• ରͷgeo_point͕શͯೖΔbounding boxΛܭࢉͯ͠ฦ٫
• geo_centroid aggregation
• ରͷgeo_point͔ΒCentroidΛܭࢉͯ͠ฦ٫
• geo_distance aggregation
• ࢦఆ͞Εͨத৺͔Βࢦఆ͞Εͨڑ͝ͱͷ݅Λฦ٫
• geo_hash aggregation
• ࢦఆ͞ΕͨprecisionͰͷgeohash͝ͱͷ݅Λฦ٫
46
Slide 47
Slide 47 text
Sorting
47
GET /_search
{
"sort" : [ {
"_geo_distance" : {
"pin.location" : [-70, 40],
"order" : "asc",
"unit" : "km",
"mode" : "min",
"distance_type" : "arc"
}
}]...
}
Slide 48
Slide 48 text
KibanaͰͷ׆༻ํ๏
Slide 49
Slide 49 text
Mapquest ...
• Ҏલ
Leaflet + MapquestͰ
ਤΛඳը
49
Slide 50
Slide 50 text
Mapquest ...
• ͦΜͳ͋Δ(2016/07)...
50
Slide 51
Slide 51 text
Elastic Tile Service
• ΄΅1िؒͰ্ཱͪ͛
51
Slide 52
Slide 52 text
Elastic Tile Service
• KibanaϢʔβͷΈ
• X-Pack
Basic licenseͰ
ΑΓζʔϜՄೳ
52
Slide 53
Slide 53 text
KibanaͷσϞ
Slide 54
Slide 54 text
ࢀߟจݙ
• Elasticsearch - The Definitive guide
‒ http://www.elastic.co/guide/en/elasticsearch/guide/current/index.html
• ॻ੶ʢຊޠʣ
‒ ElasticSearchServerຊޠ൛
‒ σʔλੳج൫ߏஙೖ
20179݄21ൃച
54
Slide 55
Slide 55 text
ࢀߟαΠτ
• Ϣʔεέʔε
• https://www.elastic.co/use-cases
• DiscussʢWebϑΥʔϥϜʣ
• https://discuss.elastic.co
• Elastic{ON}ͷϏσΦͱࢿྉ
• https://www.elastic.co/elasticon/videos
• αϙʔτϝχϡʔ
• https://www.elastic.co/subscriptions
55
Slide 56
Slide 56 text
Thanks for listening!
Q & A
We’re hiring!
https://www.elastic.co/about/careers/
We’re helping!
https://www.elastic.co/subscriptions
http://training.elastic.co