Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Where is the ghost in the ghost island? Explore by Java and Mongo

Where is the ghost in the ghost island? Explore by Java and Mongo

Su, Yung-Shun (LINE Taiwan)
TWJUG 2019 4月份聚會
https://www.meetup.com/ja-JP/taiwanjug/events/259887838/

LINE Developers

April 11, 2019
Tweet

More Decks by LINE Developers

Other Decks in Technology

Transcript

  1. Ghost Island
    Where is the ghost in the ghost island

    View full-size slide

  2. • ghost-island-analysis
    • ghost-island-batch
    • ghost-island-cargo
    • ghost-island-dashboard
    • ghost-island-gateway
    Technical Stack

    View full-size slide

  3. • Spring web
    • Spark
    • MongoDB
    ghost-island-analysis

    View full-size slide

  4. • Spring batch
    • MongoDB
    ghost-island-batch

    View full-size slide

  5. • NodeJs
    • Loopback
    • MongoDB
    ghost-island-cargo

    View full-size slide

  6. • Vuejs
    • D3js
    ghost-island-dashboard

    View full-size slide

  7. • Spring gateway
    ghost-island-gateway

    View full-size slide

  8. LINE Pay
    Merchant Map
    - List
    - Nearby
    - Search

    View full-size slide

  9. Major feature
    List, Nearby, Search

    View full-size slide

  10. Elasticsearch vs MongoDB
    Full-text search, response time, keyword length: 1
    136.34
    Elasticsearch
    13.85
    MongoDB

    View full-size slide

  11. Elasticsearch vs MongoDB
    Full-text search, response time, keyword length: 2
    99.04
    Elasticsearch
    1152.60
    MongoDB

    View full-size slide

  12. Elasticsearch vs MongoDB
    Full-text search, response time, keyword length: 3
    123.71
    Elasticsearch
    1299.93
    MongoDB

    View full-size slide

  13. ⼤大標題Title
    副標題subtitle
    Response Time (ms)
    Full text search
    1 char 2 chars 3 chars
    Elasticsearch MongoDB Elasticsearch MongoDB Elasticsearch MongoDB
    1,299.93
    1,152.6
    13.85 124
    99
    136
    Elasticsearch MongoDB

    View full-size slide

  14. Elasticsearch vs MongoDB
    Full-text search, TPS, keyword length: 1
    367.9
    Elasticsearch
    3546.9
    MongoDB

    View full-size slide

  15. Elasticsearch vs MongoDB
    Full-text search, TPS, keyword length: 2
    503.8
    Elasticsearch
    42.9
    MongoDB

    View full-size slide

  16. Elasticsearch vs MongoDB
    Full-text search , TPS, keyword length: 3
    403.5
    Elasticsearch
    1299.93
    MongoDB

    View full-size slide

  17. ⼤大標題Title
    副標題subtitle
    Transaction per second (TPS)
    Full text search
    1 char 2 chars 3 chars
    Elasticsearch MongoDB Elasticsearch MongoDB Elasticsearch MongoDB
    36.2
    42.9
    3,546.9
    404
    504
    368
    Elasticsearch MongoDB

    View full-size slide

  18. Merchant Search
    Elasticsearch
    - Good performance
    - Stable

    View full-size slide

  19. Elasticsearch vs MongoDB
    Nearby search
    Vuser
    99 Vuser
    Duration
    3/10 mins
    Radius
    10/30/50 km
    Merchants
    100 merchants

    View full-size slide

  20. Elasticsearch vs MongoDB
    Nearby search, 10km, TPS

    View full-size slide

  21. Elasticsearch vs MongoDB
    Nearby search, 30km, TPS

    View full-size slide

  22. Elasticsearch vs MongoDB
    Nearby search, 30km, TPS

    View full-size slide

  23. Nearby
    MongoDB
    - MongoDB: 2dsphere index
    - Elasticsearch: GeoPoint mapping

    View full-size slide

  24. • Geo query: MongoDB
    • Text query: Elasticsearch
    • Our choice: MongoDB
    Elasticsearch vs MongoDB

    View full-size slide

  25. Ghost Island
    A.B.C.D
    Analysis
    Spark
    Batch
    Spring
    Cargo
    NodeJS
    Dashboard
    VueJs

    View full-size slide

  26. Ghost Island
    Architecture
    OpenData
    Batch
    Spring
    Read
    Process
    Cargo
    NodeJS
    Storage
    MongoDB
    Write
    Analysis
    Spark
    Gateway
    Spring
    Text Search
    List & Nearby
    Dashboard
    VueJS

    View full-size slide

  27. @Bean
    public FlatFileItemReader reader() {
    return new FlatFileItemReaderBuilder()
    .name("openDataItemReader")
    .resource(new FileSystemResource("./data-source/raw.csv")).linesToSkip(1)
    .delimited().names(new String[] {"localTime", "description", "hurt", "type", "longitude",
    "latitude"})
    .targetType(OpenData.class).build();
    }
    發⽣生時間,發⽣生地點,死亡受傷⼈人數,⾞車車種,經度,緯度
    107年年01⽉月01⽇日 00時03分00秒,臺中市⻄西屯區上⽯石⾥里里⻄西屯路路⼆二段273之1號前0.0公尺,死亡0;受傷1,普通重型-機⾞車車;⾃自⽤用-⼩小客⾞車車,120.644124,24.174996
    107年年01⽉月01⽇日 00時04分00秒,雲林林縣林林內鄉國道3號 253公⾥里里500.0公尺處南向中線,死亡0;受傷1,⾃自⽤用-⼩小客⾞車車;⾃自⽤用-⼩小客⾞車車,120.605676,23.762712
    107年年01⽉月01⽇日 00時10分00秒,新北市泰⼭山區⼤大科⼀一路路 / 新北市泰⼭山區黎黎明路路,死亡0;受傷1,普通重型-機⾞車車,121.395695,25.062396
    107年年01⽉月01⽇日 00時13分00秒,⾼高雄市⼩小港區中⼭山四路路前 / ⾼高雄市⼩小港區中安路路,死亡0;受傷1,普通重型-機⾞車車;計程⾞車車-⼩小客⾞車車,120.346766,22.569669
    107年年01⽉月01⽇日 00時19分00秒,臺南市安南區北安路路東側 / 臺南市安南區賢安街,死亡0;受傷1,普通重型-機⾞車車,120.205404,23.026156
    107年年01⽉月01⽇日 00時20分23秒,臺南市東區中華東路路⼆二段北側 / 臺南市東區東⾨門路路,死亡0;受傷1,⾃自⽤用-⼩小客⾞車車;普通重型-機⾞車車,120.229347,22.981113
    107年年01⽉月01⽇日 00時20分27秒,宜蘭蘭縣宜蘭蘭市中⼭山路路3段12號前0.0公尺,死亡0;受傷1,⾃自⽤用-⼩小客⾞車車;普通重型-機⾞車車,121.752189,24.754909
    107年年01⽉月01⽇日 00時20分37秒,雲林林縣古坑鄉縣158⼄乙線 12公⾥里里100.0公尺處⻄西向外側,死亡0;受傷2,普通重型-機⾞車車;乘客-⼈人,120.547546,23.628280
    Batch

    View full-size slide

  28. Ghost Island
    Dashboard - List
    - Front-End (Vue.js)
    - Data visualization (D3.js)
    - Back-End (NodeJs + Loopback4 + MongoDB Connector + MongoDB)

    View full-size slide

  29. @get('/spots', {
    responses: {
    '200': {
    description: 'Array of Spot model instances',
    content: {
    'application/json': {
    schema: {type: 'array', items: {'x-ts-type': Spot}},
    },
    },
    },
    },
    })
    async find(
    @param.query.object('filter', getFilterSchemaFor(Spot)) filter?: Filter,
    ): Promise {
    return await this.spotRepository.find(filter);
    }
    Cargo

    View full-size slide

  30. fetch("/api/spots")
    .then(function(response){
    return response.json()
    })
    .then(function(json){
    //{...}
    });
    for ( let i = 0 ; i < json.length ; ++i ){
    let
    orgCoordinates=json[i].location.coordinates;
    let spot = {
    coordinates: projection(orgCoordinates)
    };
    svg.append("circle")
    .attr("fill", "rgba(255,0,0,0.1)")
    .attr("cx", spot.coordinates[0])
    .attr("cy", spot.coordinates[1])
    .attr("r", 1);
    }
    Dashboard

    View full-size slide

  31. Ghost Island
    Dashboard - Nearby
    - Front-End (Vue.js)
    - Data visualization (D3.js)
    - Back-End (NodeJs + Loopback4 + MongoDB Connector + MongoDB)

    View full-size slide

  32. fetch(“/api/spots?filter[where][location][near]="+location[0]+","+location[1]
    +"&filter[where][location][maxDistance]=10&filter[where][location][unit]=kilometers")
    .then(function(response){
    return response.json()
    })
    Dashboard

    View full-size slide

  33. Ghost Island
    Dashboard - Search
    - Front-End (Vue.js)
    - Data visualization (D3.js)
    - Back-End (Spring web + Spark + MongoDB Connector + MongoDB)

    View full-size slide

  34. fetch("/analysis/search?word="+word)
    .then(function(response){
    return response.json();
    })
    @RestController
    @RequestMapping(value="/search")
    public class SearchController {
    //{...}
    }
    Dashboard

    View full-size slide

  35. @RequestMapping(value="")
    public List> search(HttpServletRequest request){
    String word = request.getParameter("word");
    Dataset ds = rdd.toDS(SpotEntity.class);
    ds.createOrReplaceTempView("spot");
    Dataset rows = sparkSession.sql(
    "SELECT description, location FROM spot WHERE description like ‘%”+word+"%'"
    ).toJSON();
    return rows.collectAsList();
    }
    Dashboard

    View full-size slide

  36. @Bean
    public RouteLocator serviceRouteLocator(RouteLocatorBuilder builder){
    return builder.routes()
    .route("cargo_route", r -> r.path("/api/**")
    .filters(f -> f.rewritePath("^/api", ""))
    .uri("http://localhost:"+cargoPort)
    )
    .route("analysis_route", r -> r.path("/analysis/**")
    .filters(f -> f.rewritePath("^/analysis", ""))
    .uri("http://localhost:"+analysisPort)
    )
    .route("batch_route", r -> r.path("/jobLauncher")
    .uri("http://localhost:"+batchPort)
    )
    .route("dashboard_route", r -> r.path("/**")
    .uri("http://localhost:"+dashboardPort)
    )
    .build();
    }
    Gateway

    View full-size slide

  37. Ghost Island
    Put all together
    List Nearby Search

    View full-size slide