Where is the ghost in the ghost island? Explore by Java and Mongo

Where is the ghost in the ghost island? Explore by Java and Mongo

Su, Yung-Shun (LINE Taiwan)
TWJUG 2019 4月份聚會
https://www.meetup.com/ja-JP/taiwanjug/events/259887838/

53850955f15249a1a9dc49df6113e400?s=128

LINE Developers

April 11, 2019
Tweet

Transcript

  1. 13.

    ⼤大標題Title 副標題subtitle Response Time (ms) Full text search 1 char

    2 chars 3 chars Elasticsearch MongoDB Elasticsearch MongoDB Elasticsearch MongoDB 1,299.93 1,152.6 13.85 124 99 136 Elasticsearch MongoDB
  2. 17.

    ⼤大標題Title 副標題subtitle Transaction per second (TPS) Full text search 1

    char 2 chars 3 chars Elasticsearch MongoDB Elasticsearch MongoDB Elasticsearch MongoDB 36.2 42.9 3,546.9 404 504 368 Elasticsearch MongoDB
  3. 19.

    Elasticsearch vs MongoDB Nearby search Vuser 99 Vuser Duration 3/10

    mins Radius 10/30/50 km Merchants 100 merchants
  4. 24.

    • Geo query: MongoDB • Text query: Elasticsearch • Our

    choice: MongoDB Elasticsearch vs MongoDB
  5. 26.

    Ghost Island Architecture OpenData Batch Spring Read Process Cargo NodeJS

    Storage MongoDB Write Analysis Spark Gateway Spring Text Search List & Nearby Dashboard VueJS
  6. 27.

    @Bean public FlatFileItemReader<OpenData> reader() { return new FlatFileItemReaderBuilder<OpenData>() .name("openDataItemReader") .resource(new

    FileSystemResource("./data-source/raw.csv")).linesToSkip(1) .delimited().names(new String[] {"localTime", "description", "hurt", "type", "longitude", "latitude"}) .targetType(OpenData.class).build(); } 發⽣生時間,發⽣生地點,死亡受傷⼈人數,⾞車車種,經度,緯度 107年年01⽉月01⽇日 00時03分00秒,臺中市⻄西屯區上⽯石⾥里里⻄西屯路路⼆二段273之1號前0.0公尺,死亡0;受傷1,普通重型-機⾞車車;⾃自⽤用-⼩小客⾞車車,120.644124,24.174996 107年年01⽉月01⽇日 00時04分00秒,雲林林縣林林內鄉國道3號 253公⾥里里500.0公尺處南向中線,死亡0;受傷1,⾃自⽤用-⼩小客⾞車車;⾃自⽤用-⼩小客⾞車車,120.605676,23.762712 107年年01⽉月01⽇日 00時10分00秒,新北市泰⼭山區⼤大科⼀一路路 / 新北市泰⼭山區黎黎明路路,死亡0;受傷1,普通重型-機⾞車車,121.395695,25.062396 107年年01⽉月01⽇日 00時13分00秒,⾼高雄市⼩小港區中⼭山四路路前 / ⾼高雄市⼩小港區中安路路,死亡0;受傷1,普通重型-機⾞車車;計程⾞車車-⼩小客⾞車車,120.346766,22.569669 107年年01⽉月01⽇日 00時19分00秒,臺南市安南區北安路路東側 / 臺南市安南區賢安街,死亡0;受傷1,普通重型-機⾞車車,120.205404,23.026156 107年年01⽉月01⽇日 00時20分23秒,臺南市東區中華東路路⼆二段北側 / 臺南市東區東⾨門路路,死亡0;受傷1,⾃自⽤用-⼩小客⾞車車;普通重型-機⾞車車,120.229347,22.981113 107年年01⽉月01⽇日 00時20分27秒,宜蘭蘭縣宜蘭蘭市中⼭山路路3段12號前0.0公尺,死亡0;受傷1,⾃自⽤用-⼩小客⾞車車;普通重型-機⾞車車,121.752189,24.754909 107年年01⽉月01⽇日 00時20分37秒,雲林林縣古坑鄉縣158⼄乙線 12公⾥里里100.0公尺處⻄西向外側,死亡0;受傷2,普通重型-機⾞車車;乘客-⼈人,120.547546,23.628280 Batch
  7. 28.

    Ghost Island Dashboard - List - Front-End (Vue.js) - Data

    visualization (D3.js) - Back-End (NodeJs + Loopback4 + MongoDB Connector + MongoDB)
  8. 29.

    @get('/spots', { responses: { '200': { description: 'Array of Spot

    model instances', content: { 'application/json': { schema: {type: 'array', items: {'x-ts-type': Spot}}, }, }, }, }, }) async find( @param.query.object('filter', getFilterSchemaFor(Spot)) filter?: Filter, ): Promise<Spot[]> { return await this.spotRepository.find(filter); } Cargo
  9. 30.

    fetch("/api/spots") .then(function(response){ return response.json() }) .then(function(json){ //{...} }); for (

    let i = 0 ; i < json.length ; ++i ){ let orgCoordinates=json[i].location.coordinates; let spot = { coordinates: projection(orgCoordinates) }; svg.append("circle") .attr("fill", "rgba(255,0,0,0.1)") .attr("cx", spot.coordinates[0]) .attr("cy", spot.coordinates[1]) .attr("r", 1); } Dashboard
  10. 31.

    Ghost Island Dashboard - Nearby - Front-End (Vue.js) - Data

    visualization (D3.js) - Back-End (NodeJs + Loopback4 + MongoDB Connector + MongoDB)
  11. 33.

    Ghost Island Dashboard - Search - Front-End (Vue.js) - Data

    visualization (D3.js) - Back-End (Spring web + Spark + MongoDB Connector + MongoDB)
  12. 35.

    @RequestMapping(value="") public List<?> search(HttpServletRequest request){ String word = request.getParameter("word"); Dataset<SpotEntity>

    ds = rdd.toDS(SpotEntity.class); ds.createOrReplaceTempView("spot"); Dataset<String> rows = sparkSession.sql( "SELECT description, location FROM spot WHERE description like ‘%”+word+"%'" ).toJSON(); return rows.collectAsList(); } Dashboard
  13. 36.

    @Bean public RouteLocator serviceRouteLocator(RouteLocatorBuilder builder){ return builder.routes() .route("cargo_route", r ->

    r.path("/api/**") .filters(f -> f.rewritePath("^/api", "")) .uri("http://localhost:"+cargoPort) ) .route("analysis_route", r -> r.path("/analysis/**") .filters(f -> f.rewritePath("^/analysis", "")) .uri("http://localhost:"+analysisPort) ) .route("batch_route", r -> r.path("/jobLauncher") .uri("http://localhost:"+batchPort) ) .route("dashboard_route", r -> r.path("/**") .uri("http://localhost:"+dashboardPort) ) .build(); } Gateway