Where is the ghost in the ghost island? Explore by Java and Mongo

Su, Yung-Shun （LINE Taiwan）
TWJUG 2019 4月份聚會
https://www.meetup.com/ja-JP/taiwanjug/events/259887838/

Transcript

1. Ghost Island Where is the ghost in the ghost island
2. • ghost-island-analysis • ghost-island-batch • ghost-island-cargo • ghost-island-dashboard • ghost-island-gateway Technical Stack
3. • Spring web • Spark • MongoDB ghost-island-analysis
4. • Spring batch • MongoDB ghost-island-batch
5. • NodeJs • Loopback • MongoDB ghost-island-cargo
6. • Vuejs • D3js ghost-island-dashboard
7. • Spring gateway ghost-island-gateway
8. LINE Pay Merchant Map - List - Nearby - Search
9. Major feature List, Nearby, Search
10. Elasticsearch vs MongoDB Full-text search, response time, keyword length: 1 136.34 Elasticsearch 13.85 MongoDB
11. Elasticsearch vs MongoDB Full-text search, response time, keyword length: 2 99.04 Elasticsearch 1152.60 MongoDB
12. Elasticsearch vs MongoDB Full-text search, response time, keyword length: 3 123.71 Elasticsearch 1299.93 MongoDB
13. ⼤大標題Title 副標題subtitle Response Time (ms) Full text search 1 char 2 chars 3 chars Elasticsearch MongoDB Elasticsearch MongoDB Elasticsearch MongoDB 1,299.93 1,152.6 13.85 124 99 136 Elasticsearch MongoDB
14. Elasticsearch vs MongoDB Full-text search, TPS, keyword length: 1 367.9 Elasticsearch 3546.9 MongoDB
15. Elasticsearch vs MongoDB Full-text search, TPS, keyword length: 2 503.8 Elasticsearch 42.9 MongoDB
16. Elasticsearch vs MongoDB Full-text search , TPS, keyword length: 3 403.5 Elasticsearch 1299.93 MongoDB
17. ⼤大標題Title 副標題subtitle Transaction per second (TPS) Full text search 1 char 2 chars 3 chars Elasticsearch MongoDB Elasticsearch MongoDB Elasticsearch MongoDB 36.2 42.9 3,546.9 404 504 368 Elasticsearch MongoDB
18. Merchant Search Elasticsearch - Good performance - Stable
19. Elasticsearch vs MongoDB Nearby search Vuser 99 Vuser Duration 3/10 mins Radius 10/30/50 km Merchants 100 merchants
20. Elasticsearch vs MongoDB Nearby search, 10km, TPS
21. Elasticsearch vs MongoDB Nearby search, 30km, TPS
22. Elasticsearch vs MongoDB Nearby search, 30km, TPS
23. Nearby MongoDB - MongoDB: 2dsphere index - Elasticsearch: GeoPoint mapping
24. • Geo query: MongoDB • Text query: Elasticsearch • Our choice: MongoDB Elasticsearch vs MongoDB
25. Ghost Island A.B.C.D Analysis Spark Batch Spring Cargo NodeJS Dashboard VueJs
26. Ghost Island Architecture OpenData Batch Spring Read Process Cargo NodeJS Storage MongoDB Write Analysis Spark Gateway Spring Text Search List & Nearby Dashboard VueJS
27. @Bean public FlatFileItemReader<OpenData> reader() { return new FlatFileItemReaderBuilder<OpenData>() .name("openDataItemReader") .resource(new FileSystemResource("./data-source/raw.csv")).linesToSkip(1) .delimited().names(new String[] {"localTime", "description", "hurt", "type", "longitude", "latitude"}) .targetType(OpenData.class).build(); } 發⽣生時間,發⽣生地點,死亡受傷⼈人數,⾞車車種,經度,緯度 107年年01⽉月01⽇日 00時03分00秒,臺中市⻄西屯區上⽯石⾥里里⻄西屯路路⼆二段273之1號前0.0公尺,死亡0;受傷1,普通重型-機⾞車車;⾃自⽤用-⼩小客⾞車車,120.644124,24.174996 107年年01⽉月01⽇日 00時04分00秒,雲林林縣林林內鄉國道3號 253公⾥里里500.0公尺處南向中線,死亡0;受傷1,⾃自⽤用-⼩小客⾞車車;⾃自⽤用-⼩小客⾞車車,120.605676,23.762712 107年年01⽉月01⽇日 00時10分00秒,新北市泰⼭山區⼤大科⼀一路路 / 新北市泰⼭山區黎黎明路路,死亡0;受傷1,普通重型-機⾞車車,121.395695,25.062396 107年年01⽉月01⽇日 00時13分00秒,⾼高雄市⼩小港區中⼭山四路路前 / ⾼高雄市⼩小港區中安路路,死亡0;受傷1,普通重型-機⾞車車;計程⾞車車-⼩小客⾞車車,120.346766,22.569669 107年年01⽉月01⽇日 00時19分00秒,臺南市安南區北安路路東側 / 臺南市安南區賢安街,死亡0;受傷1,普通重型-機⾞車車,120.205404,23.026156 107年年01⽉月01⽇日 00時20分23秒,臺南市東區中華東路路⼆二段北側 / 臺南市東區東⾨門路路,死亡0;受傷1,⾃自⽤用-⼩小客⾞車車;普通重型-機⾞車車,120.229347,22.981113 107年年01⽉月01⽇日 00時20分27秒,宜蘭蘭縣宜蘭蘭市中⼭山路路3段12號前0.0公尺,死亡0;受傷1,⾃自⽤用-⼩小客⾞車車;普通重型-機⾞車車,121.752189,24.754909 107年年01⽉月01⽇日 00時20分37秒,雲林林縣古坑鄉縣158⼄乙線 12公⾥里里100.0公尺處⻄西向外側,死亡0;受傷2,普通重型-機⾞車車;乘客-⼈人,120.547546,23.628280 Batch
28. Ghost Island Dashboard - List - Front-End (Vue.js) - Data visualization (D3.js) - Back-End (NodeJs + Loopback4 + MongoDB Connector + MongoDB)
29. @get('/spots', { responses: { '200': { description: 'Array of Spot model instances', content: { 'application/json': { schema: {type: 'array', items: {'x-ts-type': Spot}}, }, }, }, }, }) async find( @param.query.object('filter', getFilterSchemaFor(Spot)) filter?: Filter, ): Promise<Spot[]> { return await this.spotRepository.find(filter); } Cargo
30. fetch("/api/spots") .then(function(response){ return response.json() }) .then(function(json){ //{...} }); for ( let i = 0 ; i < json.length ; ++i ){ let orgCoordinates=json[i].location.coordinates; let spot = { coordinates: projection(orgCoordinates) }; svg.append("circle") .attr("fill", "rgba(255,0,0,0.1)") .attr("cx", spot.coordinates[0]) .attr("cy", spot.coordinates[1]) .attr("r", 1); } Dashboard
31. Ghost Island Dashboard - Nearby - Front-End (Vue.js) - Data visualization (D3.js) - Back-End (NodeJs + Loopback4 + MongoDB Connector + MongoDB)
32. fetch(“/api/spots?filter[where][location][near]="+location[0]+","+location[1] +"&filter[where][location][maxDistance]=10&filter[where][location][unit]=kilometers") .then(function(response){ return response.json() }) Dashboard
33. Ghost Island Dashboard - Search - Front-End (Vue.js) - Data visualization (D3.js) - Back-End (Spring web + Spark + MongoDB Connector + MongoDB)
34. fetch("/analysis/search?word="+word) .then(function(response){ return response.json(); }) @RestController @RequestMapping(value="/search") public class SearchController { //{...} } Dashboard
35. @RequestMapping(value="") public List<?> search(HttpServletRequest request){ String word = request.getParameter("word"); Dataset<SpotEntity> ds = rdd.toDS(SpotEntity.class); ds.createOrReplaceTempView("spot"); Dataset<String> rows = sparkSession.sql( "SELECT description, location FROM spot WHERE description like ‘%”+word+"%'" ).toJSON(); return rows.collectAsList(); } Dashboard
36. @Bean public RouteLocator serviceRouteLocator(RouteLocatorBuilder builder){ return builder.routes() .route("cargo_route", r -> r.path("/api/**") .filters(f -> f.rewritePath("^/api", "")) .uri("http://localhost:"+cargoPort) ) .route("analysis_route", r -> r.path("/analysis/**") .filters(f -> f.rewritePath("^/analysis", "")) .uri("http://localhost:"+analysisPort) ) .route("batch_route", r -> r.path("/jobLauncher") .uri("http://localhost:"+batchPort) ) .route("dashboard_route", r -> r.path("/**") .uri("http://localhost:"+dashboardPort) ) .build(); } Gateway
37. Ghost Island Put all together List Nearby Search