Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Where is the ghost in the ghost island? Explore by Java and Mongo

Where is the ghost in the ghost island? Explore by Java and Mongo

Su, Yung-Shun (LINE Taiwan)
TWJUG 2019 4月份聚會
https://www.meetup.com/ja-JP/taiwanjug/events/259887838/

53850955f15249a1a9dc49df6113e400?s=128

LINE Developers
PRO

April 11, 2019
Tweet

Transcript

  1. Ghost Island Where is the ghost in the ghost island

  2. • ghost-island-analysis • ghost-island-batch • ghost-island-cargo • ghost-island-dashboard • ghost-island-gateway

    Technical Stack
  3. • Spring web • Spark • MongoDB ghost-island-analysis

  4. • Spring batch • MongoDB ghost-island-batch

  5. • NodeJs • Loopback • MongoDB ghost-island-cargo

  6. • Vuejs • D3js ghost-island-dashboard

  7. • Spring gateway ghost-island-gateway

  8. LINE Pay Merchant Map - List - Nearby - Search

  9. Major feature List, Nearby, Search

  10. Elasticsearch vs MongoDB Full-text search, response time, keyword length: 1

    136.34 Elasticsearch 13.85 MongoDB
  11. Elasticsearch vs MongoDB Full-text search, response time, keyword length: 2

    99.04 Elasticsearch 1152.60 MongoDB
  12. Elasticsearch vs MongoDB Full-text search, response time, keyword length: 3

    123.71 Elasticsearch 1299.93 MongoDB
  13. ⼤大標題Title 副標題subtitle Response Time (ms) Full text search 1 char

    2 chars 3 chars Elasticsearch MongoDB Elasticsearch MongoDB Elasticsearch MongoDB 1,299.93 1,152.6 13.85 124 99 136 Elasticsearch MongoDB
  14. Elasticsearch vs MongoDB Full-text search, TPS, keyword length: 1 367.9

    Elasticsearch 3546.9 MongoDB
  15. Elasticsearch vs MongoDB Full-text search, TPS, keyword length: 2 503.8

    Elasticsearch 42.9 MongoDB
  16. Elasticsearch vs MongoDB Full-text search , TPS, keyword length: 3

    403.5 Elasticsearch 1299.93 MongoDB
  17. ⼤大標題Title 副標題subtitle Transaction per second (TPS) Full text search 1

    char 2 chars 3 chars Elasticsearch MongoDB Elasticsearch MongoDB Elasticsearch MongoDB 36.2 42.9 3,546.9 404 504 368 Elasticsearch MongoDB
  18. Merchant Search Elasticsearch - Good performance - Stable

  19. Elasticsearch vs MongoDB Nearby search Vuser 99 Vuser Duration 3/10

    mins Radius 10/30/50 km Merchants 100 merchants
  20. Elasticsearch vs MongoDB Nearby search, 10km, TPS

  21. Elasticsearch vs MongoDB Nearby search, 30km, TPS

  22. Elasticsearch vs MongoDB Nearby search, 30km, TPS

  23. Nearby MongoDB - MongoDB: 2dsphere index - Elasticsearch: GeoPoint mapping

  24. • Geo query: MongoDB • Text query: Elasticsearch • Our

    choice: MongoDB Elasticsearch vs MongoDB
  25. Ghost Island A.B.C.D Analysis Spark Batch Spring Cargo NodeJS Dashboard

    VueJs
  26. Ghost Island Architecture OpenData Batch Spring Read Process Cargo NodeJS

    Storage MongoDB Write Analysis Spark Gateway Spring Text Search List & Nearby Dashboard VueJS
  27. @Bean public FlatFileItemReader<OpenData> reader() { return new FlatFileItemReaderBuilder<OpenData>() .name("openDataItemReader") .resource(new

    FileSystemResource("./data-source/raw.csv")).linesToSkip(1) .delimited().names(new String[] {"localTime", "description", "hurt", "type", "longitude", "latitude"}) .targetType(OpenData.class).build(); } 發⽣生時間,發⽣生地點,死亡受傷⼈人數,⾞車車種,經度,緯度 107年年01⽉月01⽇日 00時03分00秒,臺中市⻄西屯區上⽯石⾥里里⻄西屯路路⼆二段273之1號前0.0公尺,死亡0;受傷1,普通重型-機⾞車車;⾃自⽤用-⼩小客⾞車車,120.644124,24.174996 107年年01⽉月01⽇日 00時04分00秒,雲林林縣林林內鄉國道3號 253公⾥里里500.0公尺處南向中線,死亡0;受傷1,⾃自⽤用-⼩小客⾞車車;⾃自⽤用-⼩小客⾞車車,120.605676,23.762712 107年年01⽉月01⽇日 00時10分00秒,新北市泰⼭山區⼤大科⼀一路路 / 新北市泰⼭山區黎黎明路路,死亡0;受傷1,普通重型-機⾞車車,121.395695,25.062396 107年年01⽉月01⽇日 00時13分00秒,⾼高雄市⼩小港區中⼭山四路路前 / ⾼高雄市⼩小港區中安路路,死亡0;受傷1,普通重型-機⾞車車;計程⾞車車-⼩小客⾞車車,120.346766,22.569669 107年年01⽉月01⽇日 00時19分00秒,臺南市安南區北安路路東側 / 臺南市安南區賢安街,死亡0;受傷1,普通重型-機⾞車車,120.205404,23.026156 107年年01⽉月01⽇日 00時20分23秒,臺南市東區中華東路路⼆二段北側 / 臺南市東區東⾨門路路,死亡0;受傷1,⾃自⽤用-⼩小客⾞車車;普通重型-機⾞車車,120.229347,22.981113 107年年01⽉月01⽇日 00時20分27秒,宜蘭蘭縣宜蘭蘭市中⼭山路路3段12號前0.0公尺,死亡0;受傷1,⾃自⽤用-⼩小客⾞車車;普通重型-機⾞車車,121.752189,24.754909 107年年01⽉月01⽇日 00時20分37秒,雲林林縣古坑鄉縣158⼄乙線 12公⾥里里100.0公尺處⻄西向外側,死亡0;受傷2,普通重型-機⾞車車;乘客-⼈人,120.547546,23.628280 Batch
  28. Ghost Island Dashboard - List - Front-End (Vue.js) - Data

    visualization (D3.js) - Back-End (NodeJs + Loopback4 + MongoDB Connector + MongoDB)
  29. @get('/spots', { responses: { '200': { description: 'Array of Spot

    model instances', content: { 'application/json': { schema: {type: 'array', items: {'x-ts-type': Spot}}, }, }, }, }, }) async find( @param.query.object('filter', getFilterSchemaFor(Spot)) filter?: Filter, ): Promise<Spot[]> { return await this.spotRepository.find(filter); } Cargo
  30. fetch("/api/spots") .then(function(response){ return response.json() }) .then(function(json){ //{...} }); for (

    let i = 0 ; i < json.length ; ++i ){ let orgCoordinates=json[i].location.coordinates; let spot = { coordinates: projection(orgCoordinates) }; svg.append("circle") .attr("fill", "rgba(255,0,0,0.1)") .attr("cx", spot.coordinates[0]) .attr("cy", spot.coordinates[1]) .attr("r", 1); } Dashboard
  31. Ghost Island Dashboard - Nearby - Front-End (Vue.js) - Data

    visualization (D3.js) - Back-End (NodeJs + Loopback4 + MongoDB Connector + MongoDB)
  32. fetch(“/api/spots?filter[where][location][near]="+location[0]+","+location[1] +"&filter[where][location][maxDistance]=10&filter[where][location][unit]=kilometers") .then(function(response){ return response.json() }) Dashboard

  33. Ghost Island Dashboard - Search - Front-End (Vue.js) - Data

    visualization (D3.js) - Back-End (Spring web + Spark + MongoDB Connector + MongoDB)
  34. fetch("/analysis/search?word="+word) .then(function(response){ return response.json(); }) @RestController @RequestMapping(value="/search") public class SearchController

    { //{...} } Dashboard
  35. @RequestMapping(value="") public List<?> search(HttpServletRequest request){ String word = request.getParameter("word"); Dataset<SpotEntity>

    ds = rdd.toDS(SpotEntity.class); ds.createOrReplaceTempView("spot"); Dataset<String> rows = sparkSession.sql( "SELECT description, location FROM spot WHERE description like ‘%”+word+"%'" ).toJSON(); return rows.collectAsList(); } Dashboard
  36. @Bean public RouteLocator serviceRouteLocator(RouteLocatorBuilder builder){ return builder.routes() .route("cargo_route", r ->

    r.path("/api/**") .filters(f -> f.rewritePath("^/api", "")) .uri("http://localhost:"+cargoPort) ) .route("analysis_route", r -> r.path("/analysis/**") .filters(f -> f.rewritePath("^/analysis", "")) .uri("http://localhost:"+analysisPort) ) .route("batch_route", r -> r.path("/jobLauncher") .uri("http://localhost:"+batchPort) ) .route("dashboard_route", r -> r.path("/**") .uri("http://localhost:"+dashboardPort) ) .build(); } Gateway
  37. Ghost Island Put all together List Nearby Search