Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Developing microservice infrastructure for LINE Securities

Developing microservice infrastructure for LINE Securities

Kentaro Maeda
LINE Financial Development Center / SRE Team Server Side Engineer
https://linedevday.linecorp.com/2020/ja/sessions/7096
https://linedevday.linecorp.com/2020/en/sessions/7096

LINE DevDay 2020

November 27, 2020
Tweet

More Decks by LINE DevDay 2020

Other Decks in Technology

Transcript

  1. View Slide

  2. Agenda
    › About LINE Securities
    › LINE Securities system issues
    › Developing microservices platform
    › Future Prospects

    View Slide

  3. About LINE Securities

    View Slide

  4. Familiar and Easy Investment
    › Small amounts
    › Launch on 2019/8
    › Easy processes
    › Simple UI

    View Slide

  5. “Ichi-Kabu” One stock trading
    › Trade until 9:00 p.m.
    › Trade stock from one stock unit
    › Trading stocks increase to 1015

    View Slide

  6. eKYC – electronic Known your customer
    › Opening account without mailing
    › Start trading as soon as the next
    business day.

    View Slide

  7. LINE Securities services
    2019/8 “Ichi-Kabu”(One stock trading) service launched.
    2019/10 eKYC Service launched.
    2019/11 Investment trust service launched.
    2020/3 LINE FX(Foreign Exchange) launched.
    2020/5 Japanese stocks Market trading and Margin trading service launched.
    2020/6 Periodic investment trust service launched.
    2020/8 Opened accounts 300,000 Over!
    2019/12 First “Time Sale Campaign”(Stock discount sale) started.

    View Slide

  8. Scale of LINE Securities system
    ※ excludes LINE FX, external services.
    2019/8 2020/8
    Services 11 70
    Servers 65 180
    The system was built on a dedicated network in LINE's data center.

    View Slide

  9. LINE Securities system issues

    View Slide

  10. LINE Securities built on on-premise
    › We need a private network separate from other LINE’s services.
    › Complex network and dedicated connection is hampered development.
    › We cannot use Verda (Private Cloud) because it didn’t satisfy financial
    security criteria at the time.

    View Slide

  11. Complicated server setup process
    › Server setup took two weeks, even for a senior
    engineer.
    › Each service has cluster with Load Balancer,
    Domain, TLS Cert, Complex network configuration.
    › Development process was not common in each
    services.

    View Slide

  12. Ineffective use of server resources
    › Over 90% requests comes when market is opening (AM 9:00 – PM 3:00)
    › Scaling out on a per service requires a lot of servers.

    View Slide

  13. For rapid service development
    › We need common development rule.
    › We need easy server setup process.
    › We developed service platform with scalable and flexible resources.

    View Slide

  14. Developing microservices platform

    View Slide

  15. Service platform policy
    › Develop from well-known tools as building blocks.
    › Use OSS to be independent of specific infrastructure and tools.
    › Develop and Maintain with a minimal team.

    View Slide

  16. Tools we choice

    View Slide

  17. Architecture Overview
    › Sharing resources in all services
    › Load balancer in Edge server
    › Dynamic Deployment
    › All services connect directly

    View Slide

  18. Server Pool & Management Servers
    › Nomad: deploy container to server pool
    › Consul: catalog service address
    › Service: service and sidecar envoy container
    › Vault: manage secret secure
    › Control Plane: inject services address to envoy

    View Slide

  19. Docker: packaging and development rule
    › Setup server commonly with ansible
    › Keep clean server pool
    › Package binary and runtime as
    image
    › Separate developer and infra with
    minimum rule

    View Slide

  20. Nomad: Deploy service containers
    › Deploy container to available server
    following manifest
    › Container liveness check, auto recovery,
    rolling update.
    › Job: container count, image name,
    resource limit, other,,,

    View Slide

  21. Consul: Service Catalog, health check
    › Store service configuration in KVS
    › Store service dynamic address
    › Health check, alert changes quickly

    View Slide

  22. envoy & Control Plane: Service mesh
    › envoy is sidecar proxy for
    ingress/egress requests.
    › envoy configuration reloadable
    from API (xDS protocol)
    › Control Plane inject service
    address from consul to envoy.

    View Slide

  23. CI/CD Pipeline
    Versioning, git-ops, DevOps

    View Slide

  24. Result of service platform
    › Easy scale up.
    After market trading launched, traffic has more than doubled.
    10 server setup and scaling up container finished one day.
    › Improved development speed
    Developers can focus on service and don’t concern to infrastructure.
    › Effective resource using.
    Sharing multiple service in one server.
    Batch processing run on server pool in nightly.

    View Slide

  25. Future Prospects

    View Slide

  26. Storage and Monitoring issues remain
    › Log storage: over 200 GB log writes in one day
    › Complex monitoring tools: Prometheus, Zabbix, Inhouse monitoring tool,,,
    › Own running middleware: kafka, Elasticsearch, Airflow.

    View Slide

  27. Measures for improvement
    › Reorganization logging and monitoring tools.
    We are looking for new monitoring tool widely. SaaS, Cloud, OSS,,,
    › Challenge again to use Verda.
    More server scale up, management services, object storage, k8s,
    We are consulting with Verda team to satisfy security standard.

    View Slide

  28. Conclusion
    › We’ll continue to improve efficiency and automation.
    › We’ve developed flexible and scalable platform on-premise.
    › We‘ll scaling LINE Securities over million account!

    View Slide

  29. Thank you

    View Slide