Elastic stackにおける 地理情報の利用方法 #foss4gj

657aeeff3fc467567dacebf8a1ea0b23?s=47 Jun Ohtani
September 16, 2017

Elastic stackにおける 地理情報の利用方法 #foss4gj

FOSS4G Tokyo 2017での発表資料です。
Geo系の紹介になります。

公式ドキュメントはこちら。https://www.elastic.co/guide
右上の虫眼鏡マークで知りたい機能を検索してください。

657aeeff3fc467567dacebf8a1ea0b23?s=128

Jun Ohtani

September 16, 2017
Tweet

Transcript

  1. ‹#› 2017/09/16 Evangelist at Elastic Jun Ohtani @johtani Elastic stackʹ͓͚Δ

    ஍ཧ৘ใͷར༻ํ๏
  2. ‹#›

  3. ΞδΣϯμ • Elastic Stack ͱ͸ʁ • Elastic StackͰͷ஍ཧ৘ใͷػೳ • LogstashͰͷGeoIP

    • ElasticsearchͰͷҢ౓ܦ౓ରԠ • KibanaͰͷ஍ਤରԠ 3
  4. about • Me, Jun Ohtani / Technical Advocate ‒ lucene-gosenίϛολʔ

    ‒ ElasticSearch Server೔ຊޠ൛ͷ຋༁ ‒ http://blog.johtani.info
 • Elasticsearch, founded in 2012 ‒ Products: Elasticsearch, Logstash, Kibana, Beats 
 X-Pack, Elastic Cloud
 Professional services: Support & development subscriptions ‒ Trainings, Consulting, SaaS 4
  5. 5 ElasticελοΫ

  6. 6 Elastic Stack 100% Φʔϓϯιʔε ʮΤϯλʔϓϥΠζ൛ʯ͸ແ͠ όʔδϣϯ 5.0Ͱ׬શ౷Ұ

  7. 7 Logstash

  8. Logstash in 10 seconds • ϩάɾσʔλͷऩूɾ؅ཧ • ऩूɺύʔεɾՃ޻ɺૹग़ • ΦʔϓϯιʔεɿApache

    License 2.0 • Ruby app (JRuby) 8
  9. Logstash architecture 9 Input Output Filter ? ? collect and

    split alter and enrich store and visualize
  10. ઃఆɿfilter 10 filter { grok { match => { "message"

    => "%{COMBINEDAPACHELOG}" } break_on_match => false } date { match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"] locale => en } geoip { source => ["clientip"] } useragent { source => "agent" target => "useragent" } }
  11. ύʔε 11 189.120.xx.xx - - [02/Dec/2014:12:18:29 +0900] "GET /manager/html HTTP/1.1"

    404 274 "-" "Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0" {… "@timestamp": "2015-04-10T09:07:49.325Z", "clientip": "189.120.xx.xx", "ident": "-", "auth": "-", "timestamp": "02/Dec/2014:12:18:29 +0900", "verb": "GET", "request": "/manager/html", … "agent": "\"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/
  12. 12 Elasticsearch

  13. ݕࡧͱͯ͠ͷ
 Elasticsearch

  14. Elasticsearchͱ͸ʁ

  15. ϑϦʔϫʔυݕࡧ 15

  16. ߜΓࠐΈ 16

  17. ϋΠϥΠτ 17

  18. ιʔτ 18

  19. ϖʔδϯά 19

  20. ूܭ 20

  21. αδΣετ 21

  22. Elasticsearch in 10 seconds • εΩʔϚϑϦʔɺ෼ࢄυΩϡϝϯτετΞɺREST & JSON • Φʔϓϯιʔε:

    Apache License 2.0 • ઃఆͳ͠Ͱ؆୯ʹࢼ͢͜ͱ͕Մೳ • JavaͰ࣮૷ɻ֦ு΋༰қ 22
  23. σʔλొ࿥ 23 curl -XPUT localhost:9200/books/book/1 -d ' { "title" :

    "Elasticsearch - The definitive guide", "authors" : "Clinton Gormley", "started" : "2013-02-04", "pages" : 230 }'
  24. elasticsearch ͞·͟·ͳܗࣜͷσʔλͰ GeoݕࡧՄೳ
 
 Ң౓ܦ౓ɺGeoHashɺ GeoShape… GEO

  25. Ecosystem • Plugins ‒ ϓϥάΠϯʹΑΔػೳͷ௥Ճ • ΫϥΠΞϯτϥΠϒϥϦ • Java, Ruby,

    python, php, perl, javascript, .NET • Scala, clojure, go 25
  26. 26 KibanaͰՄࢹԽ

  27. Kibana 5 • ElasticsearchͷσʔλΛՄࢹԽ • Node.js server & JavaScript •

    Apache License 2.0 • Elastic Stackͷ૭ͷ໾ׂ • ༷ʑͳGUIΛPluginͱ͍ͯެ։ • MarvelɺSenseɺTimelionͳͲ 27
  28. Kibana 5 28

  29. 29 X-Pack ؆୯ʹΠϯετʔϧ Elastic StackΛ֦ு αϒεΫϦϓγϣϯʹؚ·ΕΔ Security Alerting Monitoring Reporting

    Graph Machine Learning
  30. LogstashͰͷ׆༻ํ๏

  31. ઃఆɿgeoip filter 31 filter { grok { match => {

    "message" => "%{COMBINEDAPACHELOG}" } break_on_match => false } date { match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"] locale => en } geoip { source => ["clientip"] } useragent { source => "agent" target => "useragent" } }
  32. IP͔ΒҢ౓ܦ౓ͳͲ෇༩ 32 "clientip": "124.35.xx.xx", "clientip": "124.35.xx.xx", "geoip": { "continent_name": "Asia",

    "city_name": "Tokyo", "country_iso_code": "JP", "region_name": "Tokyo", "location": { "lon": 139.7559, "lat": 35.6845
  33. geoip filter • MaxMind GeoLite2 CityσʔλϕʔεΛϥΠϒϥϦʹόϯυϧ • IPv4ͱIPv6ͷ྆ํʹରԠ • Autonomous

    System Numberͷσʔλϕʔε΋੾Γସ͑Մೳ • MaxMindͷ঎༻σʔλϕʔε΋ར༻Մೳ • `database`ύϥϝʔλͰϑΝΠϧΛࢦఆՄೳ 33
  34. ElasticsearchͰͷ஍ཧ৘ใͷ׆༻

  35. Datatype • geo_point • Ң౓ܦ౓ͷϙΠϯτ • Geo·ͨ͸ڑ཭ʹج͍ͮͨσʔλͷूܭ • ڑ཭ʹΑΔιʔτɺGeoʹج͍ͮͨؔ࿈౓ͷௐ੔ •

    geo_shape • Geo ShapeσʔλʢϙϦΰϯ΍ۣܗʣ • geohash΋͘͠͸quadtree 35
  36. Sample data Point 36 { "location" : { "type" :

    "point", "coordinates" : [-77.03653, 38.897676] } }
  37. Sample data LineString 37 { "location" : { "type" :

    "linestring", "coordinates" : [[-77.03653, 38.897676], 
 [-77.009051, 38.889939]] } }
  38. Sample data Polygon 38 { "location" : { "type" :

    "polygon", "coordinates" : [ [ [100.0, 0.0], [101.0, 0.0], [101.0, 1.0], [100.0, 1.0], [100.0, 0.0] ] ] } }
  39. Sample data Geometry Collection 39 { "location" : { "type":

    "geometrycollection", "geometries": [ { "type": "point", "coordinates": [100.0, 0.0] },{ "type": "linestring", "coordinates": [ [101.0, 0.0], [102.0, 1.0] ] } ]
  40. Queries 40 • geo_shape query • geo_shapeͷݕࡧ • geo_bounding_box query

    • geo_pointͷݕࡧʢۣܗʹΑΔݕࡧʣ • geo_distance/geo_distance_range query • geo_pointͷݕࡧʢத৺఺ʴڑ཭ʹΑΔݕࡧʣ • geo_polygon query • geo_pointͷݕࡧʢϙϦΰϯʹΑΔݕࡧʣ
  41. Queries geo_shape query 41 GET /example/_search {... "filter": { "geo_shape":

    { "location": { "shape": { "type": "envelope", "coordinates" : [[13.0, 53.0], [14.0, 52.0]] }, "relation": "within" }...
  42. Queries geo_bounding_box query 42 "filter" : { "geo_bounding_box" : {

    "pin.location" : { "top_left" : { "lat" : 40.73, "lon" : -74.1 }, "bottom_right" : { "lat" : 40.01, "lon" : -71.12 }...
  43. Queries geo_distance query 43 ... "filter" : { "geo_distance" :

    { "distance" : "200km", "pin.location" : { "lat" : 40, "lon" : -70 } } }...
  44. Queries geo_distance_range query 44 ... "filter" : { "geo_distance_range" :

    { "from" : "200km", "to" : "400km", "pin.location" : { "lat" : 40, "lon" : -70 } } }...
  45. Queries geo_polygon query 45 ... "filter" : { "geo_polygon" :

    { "person.location" : { "points" : [ {"lat" : 40, "lon" : -70}, {"lat" : 30, "lon" : -80}, {"lat" : 20, "lon" : -90} ] } }...
  46. aggregation (grouping) • geo_bounds aggregation • ର৅ͷgeo_point͕શͯೖΔbounding boxΛܭࢉͯ͠ฦ٫ • geo_centroid

    aggregation • ର৅ͷgeo_point͔ΒCentroidΛܭࢉͯ͠ฦ٫ • geo_distance aggregation • ࢦఆ͞Εͨத৺఺͔Βࢦఆ͞Εͨڑ཭͝ͱͷ݅਺Λฦ٫ • geo_hash aggregation • ࢦఆ͞ΕͨprecisionͰͷgeohash͝ͱͷ݅਺Λฦ٫ 46
  47. Sorting 47 GET /_search { "sort" : [ { "_geo_distance"

    : { "pin.location" : [-70, 40], "order" : "asc", "unit" : "km", "mode" : "min", "distance_type" : "arc" } }]... }
  48. KibanaͰͷ׆༻ํ๏

  49. Mapquest ... • Ҏલ͸
 Leaflet + MapquestͰ
 ஍ਤΛඳը 49

  50. Mapquest ... • ͦΜͳ͋Δ೔(2016/07)... 50

  51. Elastic Tile Service • ΄΅1िؒͰ্ཱͪ͛ 51

  52. Elastic Tile Service • KibanaϢʔβͷΈ • X-Pack 
 Basic licenseͰ


    ΑΓζʔϜՄೳ 52
  53. KibanaͷσϞ

  54. ࢀߟจݙ • Elasticsearch - The Definitive guide ‒ http://www.elastic.co/guide/en/elasticsearch/guide/current/index.html •

    ॻ੶ʢ೔ຊޠʣ ‒ ElasticSearchServer೔ຊޠ൛ ‒ σʔλ෼ੳج൫ߏஙೖ໳
 2017೥9݄21೔ൃച 54
  55. ࢀߟαΠτ • Ϣʔεέʔε • https://www.elastic.co/use-cases • DiscussʢWebϑΥʔϥϜʣ • https://discuss.elastic.co •

    Elastic{ON}ͷϏσΦͱࢿྉ • https://www.elastic.co/elasticon/videos • αϙʔτϝχϡʔ • https://www.elastic.co/subscriptions 55
  56. Thanks for listening! Q & A We’re hiring! https://www.elastic.co/about/careers/ We’re

    helping! https://www.elastic.co/subscriptions http://training.elastic.co