Developing microservice infrastructure for LINE Securities

Kentaro Maeda
LINE Financial Development Center / SRE Team Server Side Engineer
https://linedevday.linecorp.com/2020/ja/sessions/7096
https://linedevday.linecorp.com/2020/en/sessions/7096

Transcript

2. Agenda › About LINE Securities › LINE Securities system issues › Developing microservices platform › Future Prospects
3. About LINE Securities
4. Familiar and Easy Investment › Small amounts › Launch on 2019/8 › Easy processes › Simple UI
5. “Ichi-Kabu” One stock trading › Trade until 9:00 p.m. › Trade stock from one stock unit › Trading stocks increase to 1015
6. eKYC – electronic Known your customer › Opening account without mailing › Start trading as soon as the next business day.
7. LINE Securities services 2019/8 “Ichi-Kabu”(One stock trading) service launched. 2019/10 eKYC Service launched. 2019/11 Investment trust service launched. 2020/3 LINE FX(Foreign Exchange) launched. 2020/5 Japanese stocks Market trading and Margin trading service launched. 2020/6 Periodic investment trust service launched. 2020/8 Opened accounts 300,000 Over! 2019/12 First “Time Sale Campaign”(Stock discount sale) started.
8. Scale of LINE Securities system ※ excludes LINE FX, external services. 2019/8 2020/8 Services 11 70 Servers 65 180 The system was built on a dedicated network in LINE's data center.
9. LINE Securities system issues
10. LINE Securities built on on-premise › We need a private network separate from other LINE’s services. › Complex network and dedicated connection is hampered development. › We cannot use Verda (Private Cloud) because it didn’t satisfy financial security criteria at the time.
11. Complicated server setup process › Server setup took two weeks, even for a senior engineer. › Each service has cluster with Load Balancer, Domain, TLS Cert, Complex network configuration. › Development process was not common in each services.
12. Ineffective use of server resources › Over 90% requests comes when market is opening (AM 9:00 – PM 3:00) › Scaling out on a per service requires a lot of servers.
13. For rapid service development › We need common development rule. › We need easy server setup process. › We developed service platform with scalable and flexible resources.
14. Developing microservices platform
15. Service platform policy › Develop from well-known tools as building blocks. › Use OSS to be independent of specific infrastructure and tools. › Develop and Maintain with a minimal team.
16. Tools we choice
17. Architecture Overview › Sharing resources in all services › Load balancer in Edge server › Dynamic Deployment › All services connect directly
18. Server Pool & Management Servers › Nomad: deploy container to server pool › Consul: catalog service address › Service: service and sidecar envoy container › Vault: manage secret secure › Control Plane: inject services address to envoy
19. Docker: packaging and development rule › Setup server commonly with ansible › Keep clean server pool › Package binary and runtime as image › Separate developer and infra with minimum rule
20. Nomad: Deploy service containers › Deploy container to available server following manifest › Container liveness check, auto recovery, rolling update. › Job: container count, image name, resource limit, other,,,
21. Consul: Service Catalog, health check › Store service configuration in KVS › Store service dynamic address › Health check, alert changes quickly
22. envoy & Control Plane: Service mesh › envoy is sidecar proxy for ingress/egress requests. › envoy configuration reloadable from API (xDS protocol) › Control Plane inject service address from consul to envoy.
23. CI/CD Pipeline Versioning, git-ops, DevOps
24. Result of service platform › Easy scale up. After market trading launched, traffic has more than doubled. 10 server setup and scaling up container finished one day. › Improved development speed Developers can focus on service and don’t concern to infrastructure. › Effective resource using. Sharing multiple service in one server. Batch processing run on server pool in nightly.
25. Future Prospects
26. Storage and Monitoring issues remain › Log storage: over 200 GB log writes in one day › Complex monitoring tools: Prometheus, Zabbix, Inhouse monitoring tool,,, › Own running middleware: kafka, Elasticsearch, Airflow.
27. Measures for improvement › Reorganization logging and monitoring tools. We are looking for new monitoring tool widely. SaaS, Cloud, OSS,,, › Challenge again to use Verda. More server scale up, management services, object storage, k8s, We are consulting with Verda team to satisfy security standard.
28. Conclusion › We’ll continue to improve efficiency and automation. › We’ve developed flexible and scalable platform on-premise. › We‘ll scaling LINE Securities over million account!
29. Thank you