Elastic stackにおける 地理情報の利用方法 #foss4gj

FOSS4G Tokyo 2017での発表資料です。
Geo系の紹介になります。

公式ドキュメントはこちら。https://www.elastic.co/guide
右上の虫眼鏡マークで知りたい機能を検索してください。

Transcript

1. ‹#› 2017/09/16 Evangelist at Elastic Jun Ohtani @johtani Elastic stackʹ͓͚Δ ஍ཧ৘ใͷར༻ํ๏
2. ‹#›
3. ΞδΣϯμ • Elastic Stack ͱ͸ʁ • Elastic StackͰͷ஍ཧ৘ใͷػೳ • LogstashͰͷGeoIP • ElasticsearchͰͷҢ౓ܦ౓ରԠ • KibanaͰͷ஍ਤରԠ 3
4. about • Me, Jun Ohtani / Technical Advocate ‒ lucene-gosenίϛολʔ ‒ ElasticSearch Server೔ຊޠ൛ͷ຋༁ ‒ http://blog.johtani.info
 • Elasticsearch, founded in 2012 ‒ Products: Elasticsearch, Logstash, Kibana, Beats 
 X-Pack, Elastic Cloud
 Professional services: Support & development subscriptions ‒ Trainings, Consulting, SaaS 4
5. 5 ElasticελοΫ
6. 6 Elastic Stack 100% Φʔϓϯιʔε ʮΤϯλʔϓϥΠζ൛ʯ͸ແ͠ όʔδϣϯ 5.0Ͱ׬શ౷Ұ
7. 7 Logstash
8. Logstash in 10 seconds • ϩάɾσʔλͷऩूɾ؅ཧ • ऩूɺύʔεɾՃ޻ɺૹग़ • ΦʔϓϯιʔεɿApache License 2.0 • Ruby app (JRuby) 8
9. Logstash architecture 9 Input Output Filter ? ? collect and split alter and enrich store and visualize
10. ઃఆɿfilter 10 filter { grok { match => { "message" => "%{COMBINEDAPACHELOG}" } break_on_match => false } date { match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"] locale => en } geoip { source => ["clientip"] } useragent { source => "agent" target => "useragent" } }
11. ύʔε 11 189.120.xx.xx - - [02/Dec/2014:12:18:29 +0900] "GET /manager/html HTTP/1.1" 404 274 "-" "Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0" {… "@timestamp": "2015-04-10T09:07:49.325Z", "clientip": "189.120.xx.xx", "ident": "-", "auth": "-", "timestamp": "02/Dec/2014:12:18:29 +0900", "verb": "GET", "request": "/manager/html", … "agent": "\"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/
12. 12 Elasticsearch
13. ݕࡧͱͯ͠ͷ
 Elasticsearch
14. Elasticsearchͱ͸ʁ
15. ϑϦʔϫʔυݕࡧ 15
16. ߜΓࠐΈ 16
17. ϋΠϥΠτ 17
18. ιʔτ 18
19. ϖʔδϯά 19
20. ूܭ 20
21. αδΣετ 21
22. Elasticsearch in 10 seconds • εΩʔϚϑϦʔɺ෼ࢄυΩϡϝϯτετΞɺREST & JSON • Φʔϓϯιʔε: Apache License 2.0 • ઃఆͳ͠Ͱ؆୯ʹࢼ͢͜ͱ͕Մೳ • JavaͰ࣮૷ɻ֦ு΋༰қ 22
23. σʔλొ࿥ 23 curl -XPUT localhost:9200/books/book/1 -d ' { "title" : "Elasticsearch - The definitive guide", "authors" : "Clinton Gormley", "started" : "2013-02-04", "pages" : 230 }'
24. elasticsearch ͞·͟·ͳܗࣜͷσʔλͰ GeoݕࡧՄೳ
 
 Ң౓ܦ౓ɺGeoHashɺ GeoShape… GEO
25. Ecosystem • Plugins ‒ ϓϥάΠϯʹΑΔػೳͷ௥Ճ • ΫϥΠΞϯτϥΠϒϥϦ • Java, Ruby, python, php, perl, javascript, .NET • Scala, clojure, go 25
26. 26 KibanaͰՄࢹԽ
27. Kibana 5 • ElasticsearchͷσʔλΛՄࢹԽ • Node.js server & JavaScript • Apache License 2.0 • Elastic Stackͷ૭ͷ໾ׂ • ༷ʑͳGUIΛPluginͱ͍ͯެ։ • MarvelɺSenseɺTimelionͳͲ 27
28. Kibana 5 28
29. 29 X-Pack ؆୯ʹΠϯετʔϧ Elastic StackΛ֦ு αϒεΫϦϓγϣϯʹؚ·ΕΔ Security Alerting Monitoring Reporting Graph Machine Learning
30. LogstashͰͷ׆༻ํ๏
31. ઃఆɿgeoip filter 31 filter { grok { match => { "message" => "%{COMBINEDAPACHELOG}" } break_on_match => false } date { match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"] locale => en } geoip { source => ["clientip"] } useragent { source => "agent" target => "useragent" } }
32. IP͔ΒҢ౓ܦ౓ͳͲ෇༩ 32 "clientip": "124.35.xx.xx", "clientip": "124.35.xx.xx", "geoip": { "continent_name": "Asia", "city_name": "Tokyo", "country_iso_code": "JP", "region_name": "Tokyo", "location": { "lon": 139.7559, "lat": 35.6845
33. geoip filter • MaxMind GeoLite2 CityσʔλϕʔεΛϥΠϒϥϦʹόϯυϧ • IPv4ͱIPv6ͷ྆ํʹରԠ • Autonomous System Numberͷσʔλϕʔε΋੾Γସ͑Մೳ • MaxMindͷ঎༻σʔλϕʔε΋ར༻Մೳ • `database`ύϥϝʔλͰϑΝΠϧΛࢦఆՄೳ 33
34. ElasticsearchͰͷ஍ཧ৘ใͷ׆༻
35. Datatype • geo_point • Ң౓ܦ౓ͷϙΠϯτ • Geo·ͨ͸ڑ཭ʹج͍ͮͨσʔλͷूܭ • ڑ཭ʹΑΔιʔτɺGeoʹج͍ͮͨؔ࿈౓ͷௐ੔ • geo_shape • Geo ShapeσʔλʢϙϦΰϯ΍ۣܗʣ • geohash΋͘͠͸quadtree 35
36. Sample data Point 36 { "location" : { "type" : "point", "coordinates" : [-77.03653, 38.897676] } }
37. Sample data LineString 37 { "location" : { "type" : "linestring", "coordinates" : [[-77.03653, 38.897676], 
 [-77.009051, 38.889939]] } }
38. Sample data Polygon 38 { "location" : { "type" : "polygon", "coordinates" : [ [ [100.0, 0.0], [101.0, 0.0], [101.0, 1.0], [100.0, 1.0], [100.0, 0.0] ] ] } }
39. Sample data Geometry Collection 39 { "location" : { "type": "geometrycollection", "geometries": [ { "type": "point", "coordinates": [100.0, 0.0] },{ "type": "linestring", "coordinates": [ [101.0, 0.0], [102.0, 1.0] ] } ]
40. Queries 40 • geo_shape query • geo_shapeͷݕࡧ • geo_bounding_box query • geo_pointͷݕࡧʢۣܗʹΑΔݕࡧʣ • geo_distance/geo_distance_range query • geo_pointͷݕࡧʢத৺఺ʴڑ཭ʹΑΔݕࡧʣ • geo_polygon query • geo_pointͷݕࡧʢϙϦΰϯʹΑΔݕࡧʣ
41. Queries geo_shape query 41 GET /example/_search {... "filter": { "geo_shape": { "location": { "shape": { "type": "envelope", "coordinates" : [[13.0, 53.0], [14.0, 52.0]] }, "relation": "within" }...
42. Queries geo_bounding_box query 42 "filter" : { "geo_bounding_box" : { "pin.location" : { "top_left" : { "lat" : 40.73, "lon" : -74.1 }, "bottom_right" : { "lat" : 40.01, "lon" : -71.12 }...
43. Queries geo_distance query 43 ... "filter" : { "geo_distance" : { "distance" : "200km", "pin.location" : { "lat" : 40, "lon" : -70 } } }...
44. Queries geo_distance_range query 44 ... "filter" : { "geo_distance_range" : { "from" : "200km", "to" : "400km", "pin.location" : { "lat" : 40, "lon" : -70 } } }...
45. Queries geo_polygon query 45 ... "filter" : { "geo_polygon" : { "person.location" : { "points" : [ {"lat" : 40, "lon" : -70}, {"lat" : 30, "lon" : -80}, {"lat" : 20, "lon" : -90} ] } }...
46. aggregation (grouping) • geo_bounds aggregation • ର৅ͷgeo_point͕શͯೖΔbounding boxΛܭࢉͯ͠ฦ٫ • geo_centroid aggregation • ର৅ͷgeo_point͔ΒCentroidΛܭࢉͯ͠ฦ٫ • geo_distance aggregation • ࢦఆ͞Εͨத৺఺͔Βࢦఆ͞Εͨڑ཭͝ͱͷ݅਺Λฦ٫ • geo_hash aggregation • ࢦఆ͞ΕͨprecisionͰͷgeohash͝ͱͷ݅਺Λฦ٫ 46
47. Sorting 47 GET /_search { "sort" : [ { "_geo_distance" : { "pin.location" : [-70, 40], "order" : "asc", "unit" : "km", "mode" : "min", "distance_type" : "arc" } }]... }
48. KibanaͰͷ׆༻ํ๏
49. Mapquest ... • Ҏલ͸
 Leaflet + MapquestͰ
 ஍ਤΛඳը 49
50. Mapquest ... • ͦΜͳ͋Δ೔(2016/07)... 50
51. Elastic Tile Service • ΄΅1िؒͰ্ཱͪ͛ 51
52. Elastic Tile Service • KibanaϢʔβͷΈ • X-Pack 
 Basic licenseͰ
 ΑΓζʔϜՄೳ 52
53. KibanaͷσϞ
54. ࢀߟจݙ • Elasticsearch - The Definitive guide ‒ http://www.elastic.co/guide/en/elasticsearch/guide/current/index.html • ॻ੶ʢ೔ຊޠʣ ‒ ElasticSearchServer೔ຊޠ൛ ‒ σʔλ෼ੳج൫ߏஙೖ໳
 2017೥9݄21೔ൃച 54
55. ࢀߟαΠτ • Ϣʔεέʔε • https://www.elastic.co/use-cases • DiscussʢWebϑΥʔϥϜʣ • https://discuss.elastic.co • Elastic{ON}ͷϏσΦͱࢿྉ • https://www.elastic.co/elasticon/videos • αϙʔτϝχϡʔ • https://www.elastic.co/subscriptions 55
56. Thanks for listening! Q & A We’re hiring! https://www.elastic.co/about/careers/ We’re helping! https://www.elastic.co/subscriptions http://training.elastic.co