Where is the ghost in the ghost island? Explore by Java and Mongo

Transcript

1. Ghost Island Where is the ghost in the ghost island

2. • ghost-island-analysis • ghost-island-batch • ghost-island-cargo • ghost-island-dashboard • ghost-island-gateway

   Technical Stack

3. • Spring web • Spark • MongoDB ghost-island-analysis

4. • Spring batch • MongoDB ghost-island-batch

5. • NodeJs • Loopback • MongoDB ghost-island-cargo

6. • Vuejs • D3js ghost-island-dashboard

7. • Spring gateway ghost-island-gateway

8. LINE Pay Merchant Map - List - Nearby - Search

9. Major feature List, Nearby, Search

10. Elasticsearch vs MongoDB Full-text search, response time, keyword length: 1

    136.34 Elasticsearch 13.85 MongoDB

11. Elasticsearch vs MongoDB Full-text search, response time, keyword length: 2

    99.04 Elasticsearch 1152.60 MongoDB

12. Elasticsearch vs MongoDB Full-text search, response time, keyword length: 3

    123.71 Elasticsearch 1299.93 MongoDB

13. ⼤大標題Title 副標題subtitle Response Time (ms) Full text search 1 char

    2 chars 3 chars Elasticsearch MongoDB Elasticsearch MongoDB Elasticsearch MongoDB 1,299.93 1,152.6 13.85 124 99 136 Elasticsearch MongoDB

14. Elasticsearch vs MongoDB Full-text search, TPS, keyword length: 1 367.9

    Elasticsearch 3546.9 MongoDB

15. Elasticsearch vs MongoDB Full-text search, TPS, keyword length: 2 503.8

    Elasticsearch 42.9 MongoDB

16. Elasticsearch vs MongoDB Full-text search , TPS, keyword length: 3

    403.5 Elasticsearch 1299.93 MongoDB

17. ⼤大標題Title 副標題subtitle Transaction per second (TPS) Full text search 1

    char 2 chars 3 chars Elasticsearch MongoDB Elasticsearch MongoDB Elasticsearch MongoDB 36.2 42.9 3,546.9 404 504 368 Elasticsearch MongoDB

18. Merchant Search Elasticsearch - Good performance - Stable

19. Elasticsearch vs MongoDB Nearby search Vuser 99 Vuser Duration 3/10

    mins Radius 10/30/50 km Merchants 100 merchants

20. Elasticsearch vs MongoDB Nearby search, 10km, TPS

21. Elasticsearch vs MongoDB Nearby search, 30km, TPS

22. Elasticsearch vs MongoDB Nearby search, 30km, TPS

23. Nearby MongoDB - MongoDB: 2dsphere index - Elasticsearch: GeoPoint mapping

24. • Geo query: MongoDB • Text query: Elasticsearch • Our

    choice: MongoDB Elasticsearch vs MongoDB

25. Ghost Island A.B.C.D Analysis Spark Batch Spring Cargo NodeJS Dashboard

    VueJs

26. Ghost Island Architecture OpenData Batch Spring Read Process Cargo NodeJS

    Storage MongoDB Write Analysis Spark Gateway Spring Text Search List & Nearby Dashboard VueJS

27. @Bean public FlatFileItemReader<OpenData> reader() { return new FlatFileItemReaderBuilder<OpenData>() .name("openDataItemReader") .resource(new

    FileSystemResource("./data-source/raw.csv")).linesToSkip(1) .delimited().names(new String[] {"localTime", "description", "hurt", "type", "longitude", "latitude"}) .targetType(OpenData.class).build(); } 發⽣生時間,發⽣生地點,死亡受傷⼈人數,⾞車車種,經度,緯度 107年年01⽉月01⽇日 00時03分00秒,臺中市⻄西屯區上⽯石⾥里里⻄西屯路路⼆二段273之1號前0.0公尺,死亡0;受傷1,普通重型-機⾞車車;⾃自⽤用-⼩小客⾞車車,120.644124,24.174996 107年年01⽉月01⽇日 00時04分00秒,雲林林縣林林內鄉國道3號 253公⾥里里500.0公尺處南向中線,死亡0;受傷1,⾃自⽤用-⼩小客⾞車車;⾃自⽤用-⼩小客⾞車車,120.605676,23.762712 107年年01⽉月01⽇日 00時10分00秒,新北市泰⼭山區⼤大科⼀一路路 / 新北市泰⼭山區黎黎明路路,死亡0;受傷1,普通重型-機⾞車車,121.395695,25.062396 107年年01⽉月01⽇日 00時13分00秒,⾼高雄市⼩小港區中⼭山四路路前 / ⾼高雄市⼩小港區中安路路,死亡0;受傷1,普通重型-機⾞車車;計程⾞車車-⼩小客⾞車車,120.346766,22.569669 107年年01⽉月01⽇日 00時19分00秒,臺南市安南區北安路路東側 / 臺南市安南區賢安街,死亡0;受傷1,普通重型-機⾞車車,120.205404,23.026156 107年年01⽉月01⽇日 00時20分23秒,臺南市東區中華東路路⼆二段北側 / 臺南市東區東⾨門路路,死亡0;受傷1,⾃自⽤用-⼩小客⾞車車;普通重型-機⾞車車,120.229347,22.981113 107年年01⽉月01⽇日 00時20分27秒,宜蘭蘭縣宜蘭蘭市中⼭山路路3段12號前0.0公尺,死亡0;受傷1,⾃自⽤用-⼩小客⾞車車;普通重型-機⾞車車,121.752189,24.754909 107年年01⽉月01⽇日 00時20分37秒,雲林林縣古坑鄉縣158⼄乙線 12公⾥里里100.0公尺處⻄西向外側,死亡0;受傷2,普通重型-機⾞車車;乘客-⼈人,120.547546,23.628280 Batch

28. Ghost Island Dashboard - List - Front-End (Vue.js) - Data

    visualization (D3.js) - Back-End (NodeJs + Loopback4 + MongoDB Connector + MongoDB)

29. @get('/spots', { responses: { '200': { description: 'Array of Spot

    model instances', content: { 'application/json': { schema: {type: 'array', items: {'x-ts-type': Spot}}, }, }, }, }, }) async find( @param.query.object('filter', getFilterSchemaFor(Spot)) filter?: Filter, ): Promise<Spot[]> { return await this.spotRepository.find(filter); } Cargo

30. fetch("/api/spots") .then(function(response){ return response.json() }) .then(function(json){ //{...} }); for (

    let i = 0 ; i < json.length ; ++i ){ let orgCoordinates=json[i].location.coordinates; let spot = { coordinates: projection(orgCoordinates) }; svg.append("circle") .attr("fill", "rgba(255,0,0,0.1)") .attr("cx", spot.coordinates[0]) .attr("cy", spot.coordinates[1]) .attr("r", 1); } Dashboard

31. Ghost Island Dashboard - Nearby - Front-End (Vue.js) - Data

    visualization (D3.js) - Back-End (NodeJs + Loopback4 + MongoDB Connector + MongoDB)

32. fetch(“/api/spots?filter[where][location][near]="+location[0]+","+location[1] +"&filter[where][location][maxDistance]=10&filter[where][location][unit]=kilometers") .then(function(response){ return response.json() }) Dashboard

33. Ghost Island Dashboard - Search - Front-End (Vue.js) - Data

    visualization (D3.js) - Back-End (Spring web + Spark + MongoDB Connector + MongoDB)

34. fetch("/analysis/search?word="+word) .then(function(response){ return response.json(); }) @RestController @RequestMapping(value="/search") public class SearchController

    { //{...} } Dashboard

35. @RequestMapping(value="") public List<?> search(HttpServletRequest request){ String word = request.getParameter("word"); Dataset<SpotEntity>

    ds = rdd.toDS(SpotEntity.class); ds.createOrReplaceTempView("spot"); Dataset<String> rows = sparkSession.sql( "SELECT description, location FROM spot WHERE description like ‘%”+word+"%'" ).toJSON(); return rows.collectAsList(); } Dashboard

36. @Bean public RouteLocator serviceRouteLocator(RouteLocatorBuilder builder){ return builder.routes() .route("cargo_route", r ->

    r.path("/api/**") .filters(f -> f.rewritePath("^/api", "")) .uri("http://localhost:"+cargoPort) ) .route("analysis_route", r -> r.path("/analysis/**") .filters(f -> f.rewritePath("^/analysis", "")) .uri("http://localhost:"+analysisPort) ) .route("batch_route", r -> r.path("/jobLauncher") .uri("http://localhost:"+batchPort) ) .route("dashboard_route", r -> r.path("/**") .uri("http://localhost:"+dashboardPort) ) .build(); } Gateway

37. Ghost Island Put all together List Nearby Search