A Journey to Developing an In-House Payment System / Spring One Platform 2019

Transcript

1. A Journey to Developing an In-House Payment System Cloud-Native Development with Spring and Pivotal Platform October 7–10, 2019 Austin Convention Center Junya Suzuki (@suzukij), Daichi Kimura, SB Payment Service #springone @s1p
2. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ About me 2  Part of the SoftBank Group in Japan. Application Engineer Junya Suzuki @suzukij Application engineer focusing on Java and Web systems. Developing new services with the Springframework and improving system operations. Platform Engineer Daichi Kimura Responsible for operational support, promotion and management of Cloud IaaS, PaaS and monitoring platform for payment services.
3. We issue credit cards "Softbank Card" to consumers. Credit Card Issuer Payment Aggregator We provide a comprehensive payment platform that offers various online payment solutions. Credit Card Acquirer We are the only payment aggregator in Japan who accepts and processes transactions made with major brands (VISA/ mastercard/ UnionPay). Softbank customers can also pay for online purchases via their phone bill as Japan’s leading carrier company. Carrier Billing Provider The Largest Payment Service Provider in Japan Our Business
4. We issue credit cards "Softbank Card" to consumer. Credit Card Issuer Payment Aggregator We provide a comprehensive payment platform that offers various online payment solutions. Credit Card Acquirer We are an only payment aggregator in Japan that accepts and processes transactions made with a major brands(VISA/ mastercard/ UnionPay) as an acquirer. We provide consumers to pay for online purchases with their phone bill as Japanese leading carrier company. Carrier Billing Provider The Largest Payment Service Provider in Japan Our Business
5. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ If you have any questions, we’d be happy to answer them at the end of our presentation
6. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Agenda • The road to the in-house payment system • Why we chose Pivotal Platform • What we achieved
7. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ The road to the in-house system
8. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ As of 2016 ... We left all development of our services to outside vendors. There were zero in-house engineers writing code. The development environment was not ready.
9. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 2016: Made operations more efficient by ourselves Problem Many mistakes due to manual operations Team
10. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 2016: Made operations more efficient by ourselves Spring Boot Selenium / Selenide Problem Developed Support Tools Introduced Tools Many mistakes due to manual operations Automate operational tasks Prepared the development environment Team
11. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 2016: Made operations more efficient by ourselves Problem Many mistakes due to manual operations Team Spring Boot Selenium / Selenide Developed Support Tools Introduced Tools Automate operational tasks 3 Engineers joined! Accelerated KAIZEN tasks by a team
12. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 2017: ① Improved service monitorings Could not grasp and share the service status quickly. Problem Team
13. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 2017: ① Improved service monitorings Could not grasp and share the service status quickly. Problem Created dashboards Introduced Tools Elasticsearch Logstash Kibana Team
14. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 2017: ② Started support for development projects Problem Legacy architectures Expensive develop/release cost Hard to monitor Team
15. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 2017: ② Started support for development projects Problem Re-architected with Spring Introduced Tools Modern software development 　Spring Boot 　Spring Cloud Legacy architectures Expensive develop/release cost Hard to monitor Support for proper CI flow Team
16. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 2017: ② Started support for development projects Problem Re-architected with Spring Introduced Tools Modern software development 　Spring Boot 　Spring Cloud Legacy architectures Expensive develop/release cost Hard to monitor Team One more engineer joined!
17. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 2016 - 2017 • Established an engineering team • Developed in-house tools for improved operations • Joined development projects for support 2018 In-house payment system project has finally started
18. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Merchants Financial Institutions E-Commerce Gaming Education RealEstate Etc E-Books/Movies Unified Payment Service Tickets Provides several payment methods as API to EC sites Credit Card Mobile Carrier Convenience Store Prepaid Card Account Transfer Point Account Integration 当社 Our Company API Development target Online Payment Service
19. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Merchants Financial Institutions E-Commerce Gaming Education RealEstate Etc E-Books/Movies Unified Payment Service Tickets Provides several payment methods as API to EC sites Credit Card Mobile Carrier Convenience Store Prepaid Card Account Transfer Point Account Integration 当社 Our Company API Development target Online Payment Service Number of adoption 111,742 stores (As of May 2019) Transaction Amount $ 28 billion (As of 2018)
20. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Merchants Financial Institutions E-Commerce Game Education RealEstate Etc E-Book/Movie Unified Payment Service Ticket Provides several payment methods as API to EC sites Credit Card Mobile Carrier Convenience Store Prepaid Card Account Transfer Point Account Integration 当社 Our Company API Development target Online Payment Service 40+ payment methods supported
21. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Merchants Financial Institutions E-Commerce Gaming Education RealEstate Etc E-Books/Movies Unified Payment Service Tickets Provides several payment methods as API to EC sites Credit Card Mobile Carrier Convenience Store Prepaid Card Account Transfer Point Account Integration 当社 Our Company API Development target Online Payment Service Located between merchant systems and financial institution systems
22. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 2018: Developing a In-house system Requirements for the new system • Development and Release with speed • Continuous improvement cycle • Observable, fault-tolerant and resilient Before… Every project was lead by external vendors （A long path from estimation / contract / requirement definition to acceptance)
23. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 2018: Developing a In-house system Requirements for the new system • Development and Release with speed • Continuous improvement cycle • Observable, fault-tolerant and resilient 今までは… 案件毎に開発ベンダさんのチカラを借りて構築 （見積もり/要件定義から検収まで長い道のり） Outsourcing made it impossible to deliver incrementally and quickly in the agile way.
24. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Requirements for the new system • Development and Release with speed • Continuous improvement cycle • Observable, fault-tolerant and resilient Speedy delivery and Continuous improvement through in-house development 2018: Developing a In-house system
25. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Team In-house dev PJ One more engineer joined! 6-person team 2018: In-house Payment System
26. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 2018: In-house Payment System Introduced Tools Cloud Native Platform based on Pivotal Platform
27. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Why we chose Pivotal Platform
28. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ ・Improvements in the release process ・Full utilization of the cloud Requirements for a Platform
29. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Why not Kubernetes？？ As an engineer, k8s is exciting, but ... ・High learning and maintenance costs ・Time required to build a “production-ready” platform ・Insufficient resources to handle k8s OR PaaS
30. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Why not Kubernetes？？ OR PaaS What we needed Needed a PROVEN platform that ・makes developers focus on application development ・enables deploying applications quickly
31. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ What we finally chose ... PaaS • Great integration with Java/Spring • Support for introducing/operating the platform • cf push experience to deploy apps
32. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ What we finally chose ... PaaS • Great integration with Java/Spring • Support for introducing/operating the platform • cf push experience to deploy apps Buildpack converts source code to a container image. Developers don’t need to write a Dockerfile. cf push + Buildpack reduces extra works!
33. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Team structure and responsibility boundary
34. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Team structure and responsibility boundary Team structure and responsibility boundary Networking Storage Servers Virtualization O/S Middleware Runtime Platform Operators 2 people
35. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Team structure and responsibility boundary Networking Storage Servers Virtualization O/S Middleware Runtime Data Application Application Developers 4 people Platform Operators 2 people
36. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Networking Storage Servers Virtualization O/S Middleware Runtime Data Application Focus on building and operating platforms Application Developers 4 people Platform Operators 2 people Team structure and responsibility boundary
37. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Networking Storage Servers Virtualization O/S Middleware Runtime Data Application Focus on design and implementation of business code Team structure and responsibility boundary Application Developers 4 people Platform Operators 2 people
38. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Networking Storage Servers Virtualization O/S Middleware Runtime Data Application 12 Factor App The only contract is “12 Factor App”. No vendor lock in. Application Developers 4 people Team structure and responsibility boundary Platform Operators 2 people
39. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ What we achieved ➢ Platform / Overall Architecture ➢ Architecture ➢ CI/CD Pipelines ➢ Observability
40. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ syslog+TLS Logstash Elasticsearch Kibana cf push Concourse Prometheus Grafana git push Architecture Overview cf create-service cf bind-service
41. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ What we achieved ➢ Platform / Overall Architecture ➢ Architecture ➢ CI/CD Pipelines ➢ Observability
42. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Application Architecture（Merchants ➡ Financial Institutions） API Gateway Service A Service B Service C Merchant X Merchant Y Merchant Z Financial Institution A Financial Institution B Financial Institution C
43. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Application Architecture（Merchants ➡ Financial Institutions） API Gateway Service A Service B Service C Merchant X Merchant Y Merchant Z Financial Institution A Financial Institution B Financial Institution C EC Sites Shopping、Games、Books、 Tickets、Real Estate and so on
44. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Application Architecture（Merchants ➡ Financial Institutions） API Gateway Service A Service B Service C Financial Institution A Financial Institution B Financial Institution C New Payment System Merchant X Merchant Y Merchant Z
45. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Application Architecture（Merchants ➡ Financial Institutions） API Gateway Service A Service B Service C Financial Institution A Financial Institution B Financial Institution C Merchant X Merchant Y Merchant Z Financial Institutions Credit Cards、Convenience Stores、 Mobile Carriers、Prepaid Cards and so on
46. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Application Architecture（Merchants ➡ Financial Institutions） API Gateway Service A Service B Service C Merchant X Financial Institution A Financial Institution B Financial Institution C Merchant X Merchant Y Merchant Z Each app is deployed on PAS as a microservice
47. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Application Architecture（Merchants ➡ Financial Institutions） API Gateway Service A Service B Service C Financial Institution A Financial Institution B Financial Institution C Merchant X Merchant Y Merchant Z Each app is implemented with Java and Spring Boot
48. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Application Architecture（Merchants ➡ Financial Institutions） API Gateway Service A Service B Service C Financial Institution A Financial Institution B Financial Institution C Merchant X Merchant Y Merchant Z API Gateway Routing layer to a Service that implements a business logic per financial institution
49. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ API Gateway Service A Service B Service C Financial Institution A Financial Institution B Financial Institution C Application Structure（ Merchant ➡ Settlement Org） Merchant X Merchant Y Merchant Z
50. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ API Gateway Service A Service B Service C Merchant X Merchant Y Merchant Z Financial Institution A Financial Institution B Financial Institution C The systems of merchants system and financial institutions are out of our control Application Structure（ Merchant ➡ Settlement Org）
51. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Hystrix API Gateway Service A Service B Service C Merchant A Merchant Y Merchant Z Financial Institution A Financial Institution B Financial Institution C Introduced Hystrix Circuit Breaker for inter-system communications Hystrix Hystrix Hystrix Application Structure（ Merchant ➡ Settlement Org）
52. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ API Gateway Service A Service B Service C Merchant X Merchant Y Merchant Z Financial Institution A Financial Institution B Financial Institution C Without Circuit Breaker, If a system outage happens in financial institution A … Application Structure（ Merchant ➡ Settlement Org）
53. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ API Gateway Service A Service B Service C Merchant X Merchant Y Merchant Z Financial Institution A Financial Institution B Financial Institution C Slow Response, Timeout Application Structure（ Merchant ➡ Settlement Org）
54. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Application Structure（ Merchant ➡ Settlement Org） API Gateway Service A Service B Service C Merchant X Merchant Y Merchant Z Financial Institution A Financial Institution B Financial Institution C The failure is propagated to Service A, blocking processes and causing possible thread exhaustion
55. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ API Gateway Service A Service B Service C Merchant X Merchant Y Merchant Z Financial Institution A Financial Institution B Financial Institution C Failure propagated to API Gateway causing blocked processes, thread depletion Application Structure（ Merchant ➡ Settlement Org）
56. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ API Gateway Service A Service B Service C Merchant X Merchant Y Merchant Z Financial Institution A Financial Institution B Financial Institution C Application Structure（ Merchant ➡ Settlement Org） Failure propagated to API Gateway causing blocked processes, thread depletion
57. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ API Gateway Service A Service B Service C Merchant X Merchant Y Merchant Z Financial Institution A Financial Institution B Financial Institution C B and C are affected by the failure of financial institution A. Application Structure（ Merchant ➡ Settlement Org）
58. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ API Gateway Service A Service B Service C Merchant X Merchant Y Merchant Z Financial Institution A Financial Institution B Financial Institution C Hystrix Hystrix Hystrix Hystrix Application Structure（ Merchant ➡ Settlement Org） With Circuit Breaker If a system outage happens in financial institution A …
59. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ API Gateway Service A Service B Service C Merchant X Merchant Y Merchant Z Financial Institution A Financial Institution B Financial Institution C Hystrix Hystrix Hystrix Circuit Breaker prevents the failure propagation. No worry about the effect to other financial institutions. Application Structure（ Merchant ➡ Settlement Org）
60. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ API Gateway Service A Service B Service C Merchant X Merchant Y Merchant Z Financial Institution A Financial Institution B Financial Institution C Hystrix Hystrix Hystrix Circuit Breaker adds fault tolerance and resiliency to the app Application Structure（ Merchant ➡ Settlement Org）
61. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Notification Gateway Receiver A Receiver B Receiver C Merchant X Merchant Y Merchant Z Financial Institution A Financial Institution B Financial Institution C Hystrix Hystrix Hystrix Hystrix Application Structure（ Settlement Org ➡ Merchant）
62. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Application Structure（ Settlement Org ➡ Merchant） Notification Gateway Receiver A Receiver B Receiver C Merchant X Merchant Y Merchant Z Financial Institution A Financial Institution B Financial Institution C Hystrix Hystrix Hystrix Hystrix
63. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Application Structure（ Settlement Org ➡ Merchant） Notification Gateway Receiver A Receiver B Receiver C Merchant X Merchant Y Merchant Z Financial Institution A Financial Institution B Financial Institution C Hystrix Hystrix Hystrix Hystrix Introduced RabbitMQ + Spring Cloud Stream for async processing
64. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Application Structure（ Settlement Org ➡ Merchant） Notification Gateway Receiver A Receiver B Receiver C Merchant X Merchant Y Merchant Z Financial Institution A Financial Institution B Financial Institution C Hystrix Hystrix Hystrix Hystrix
65. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Application Structure（ Settlement Org ➡ Merchant） Notification Gateway Receiver A Receiver B Receiver C Merchant X Merchant Y Merchant Z Financial Institution A Financial Institution B Financial Institution C Hystrix Hystrix Hystrix Hystrix When a failure happens in the merchant system
66. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Application Structure（ Settlement Org ➡ Merchant） Notification Gateway Receiver A Receiver B Receiver C Merchant X Merchant Y Merchant Z Financial Institution A Financial Institution B Financial Institution C Hystrix Hystrix Hystrix Hystrix The message will be diverted to a “Dead Letter Queue” and requeued later
67. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Application Structure（ Settlement Org ➡ Merchant） Notification Gateway Receiver A Receiver B Receiver C Merchant X Merchant Y Merchant Z Financial Institution A Financial Institution B Financial Institution C Hystrix Hystrix Hystrix Even if an outage happens in the merchant, the Circuit Breaker will prevent the failure propagation. Hystrix
68. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Current state of the production environment
69. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Current state of the production environment Total Applications(Services) 13 ⤴
70. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Current state of the production environment Total Instances 47 ⤴
71. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ What we achieved ➢ Platform / Overall Architecture ➢ Architecture ➢ CI/CD Pipelines ➢ Observability
72. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ CI / CD Pipeline https://concourse-ci.org/ Concourse
73. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ https://concourse-ci.org/ Concourse CI / CD Pipeline
74. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Concourse - Dashboard
75. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Concourse Pipeline from dev to prod Staging Develop Production Push to a branch triggers the pipeline
76. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Concourse Pipeline from dev to prod Staging Develop Production
77. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Unit test for each Java version to detect issues on new versions earlier Concourse Pipeline from dev to prod Staging Develop Production
78. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Concourse Pipeline from dev to prod Staging Develop Production
79. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Concourse Pipeline from dev to prod Staging Develop Production
80. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Develop Cycle by CI Concourse Pipeline from dev to prod Staging Develop Production
81. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Concourse Pipeline from dev to prod Staging Develop Production
82. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Merging to master branch triggers deploy to Nexus repository Concourse Pipeline from dev to prod Staging Develop Production
83. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Production One-click Deploy to production from Nexus Repository using nexus concourse resource Concourse Pipeline from dev to prod
84. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Testing with API Gateway Service A Service B Service C Financial Institution B Mock Financial Institution B Mock Financial Institution B Mock
85. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Testing with • Load test • Soak test
86. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Load Test Problem Cause HTTP transmission processing is low throughput PoolingHttpClientConnectionManager pool setting mistake SQL execution processing low throughput HikariCP DB connection pool setting mistake Unexpected short of Circuit Breaker Hystrix tuning mistakes RabbitMQ publish stopped Memory exhaustion stops queuing messages Throughput 20X Service Down 0
87. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Soak Test (1 week, 168 hours)
88. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Load test and E2E test by During development, JMeter tests are automatically executed every day. Notify Slack of report screenshot. Report html cf push
89. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Load test and E2E test by During development, JMeter tests are automatically executed every day. Notify Slack of report screenshot. Performance testing should be repeated over and over during development to ensure continuous improvement.
90. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ What we achieved ➢ Observability ◦ Metrics ◦ Logging ◦ Tracing
91. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ What is overvability? ※Metrics, tracing, and logging　 https://peter.bourgon.org/blog/2017/02/21/metrics-tracing-and-logging.html Metrics Logging Tracing Make distributed system easy to measure
92. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Our Observability choices
93. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ What we achieved ➢ Observability ◦ Metrics ◦ Logging ◦ Tracing
94. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Metrics Visualizing Metrics and Alertings Prometheus ・BOSH(VM) Metrics ・CF Metrics ・App Metrics ・etc.
95. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Metrics - Prometheus 　Using Prometheus BOSH Release (OSS) https://github.com/bosh-prometheus/prometheus-boshrelease Prometheus Web UI Grafana Preset Dashboard Preset Alerts
96. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ rabbitmq Monitor all BOSH VMs No need to configure per instance ⇒ Eliminate extra tasks, missed monitorings { "targets":[ "10.0.1.1", "10.0.2.1", "10.0.3.1", "10.0.1.2", "10.0.2.2", ], "labels":{ "__meta_bosh_deployment":"rabbitmq", "__meta_bosh_job_process_name":"rabbitmq-server" } } Monitor automatically Check deployments Scale out Targets are added automatically BOSH Director
97. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Org A Monitor all Micrometer Metrics through Promregator No need to configure per app ⇒ Eliminate extra tasks, missed monitorings Collect Metrics per app Collect Metrics for all apps Promregator SpaceA cf push CF Cloud Controller Check apps Devs don’t care Prometheus Prometheus Aggregator for Cloud Foundry https://github.com/promregator/promregator Monitor all Apps Considering migration to Metrics Registrar
98. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Monitor Apps with Micrometer Collect and Expose Metrics Micrometer <dependency> <groupId>io.micrometer</groupId> <artifactId>micrometer-registry-prometheus</artifactId> </dependency> management: endpoints: web: exposure: include: "*" base-path: /actuator
99. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Grafana Dashboard Also shows past changes Visualize if each deployment is healthy at this moment Visualize if each VM is healthy at this moment Warning Ops BOSH VM’s status and load Warning
100. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Grafana Dashboard Ops RabbitMQ status・throughput CF Router status・throughput 30+ dashboards
101. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Grafana Dashboard Dev Apps Metrics We can always check all metrics in Grafana that we have ever collected from logs ・CPU ・JVM Memory（per area） ・Thread ・GC（frequency、time） ・Classloader
102. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Grafana Dashboard Dev Apps Metrics Can always check metrics in Grafana we have ever collected from logs ・CPU ・JVM Memory（per area） ・Thread ・GC（frequency、time） ・Classloader Example: GC Metrics
103. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Alerting Despite lots of dashboards Action is triggered by alerting
104. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Alertmanager Ops Dev No outsourcing to external monitoring center org_name: OrgA severity: fatal org_name: OrgA severity: /.*/ severity: fatal severity: /.*/ Set route/receiver using org_name, severity as keys Twilio call for emergency (24 hour support) Alert system we developed
105. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Link to the Grafana dashboard Alerting Example Detect RabbitMQ Dead Letter Queue Call
106. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Custom Alerts (other than presets) Add custom alerts using BOSH Operations-file ・BOSH VM CPU usage (Preset includes only load average) ・CF app container metrics (CPU・RAM・DISK) ・CF memory usage per Org ・Circuit breaker status ・RabbitMQ DLQ ・RabbitMQ queue depth ・etc. Detect the decrease of the service level before user’s report
107. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ What we achieved ➢ Observability ◦ Metrics ◦ Logging ◦ Tracing
108. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Logging Aggregate and Visualize Platform/App Logs using Elastic Stack Log Alerting 　・ElastAlert 　・Praeco
109. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Logging Aggregate and Visualize Platform/App Logs using Elastic Stack
110. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Logging - BOSH Elastic Stack Using Elastic Stack BOSH Release (OSS) Search Logs Ops Dev Biz ElastAlert Praeco Alerting Firehose to syslog Syslog + TLS Syslog + TLS Store Logs Apps Platform Elastic Stack BOSH Release https://github.com/bosh-elastic-stack/elastic-stack-bosh-deployment
111. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Store Logs Firehose-to -syslog Syslog + TLS Apps only need to output logs to stdout Logs are collected by Platform App Logs - aggregated by Firehose-to-syslog Firehose-to -syslog Firehose-to -syslog Loggregator Agent Doppler STD OUT STD OUT Collect Logs Firehose to syslog https://github.com/cloudfoundry-community/firehose-to-syslog Considering migration to Syslog Drain
112. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Store Logs Syslog + TLS Using BOSH Runtime-Config to apply to all deployments No Need to configure where to forward logs per deployment Platform Logs --- releases: - name: syslog version: "11.4.0" url: "https://bosh.io/xxx?v=11.4.0" sha1: "xxx" addons: - name: syslog exclude: jobs: - name: syslog_forwarder release: syslog jobs: - name: syslog_forwarder release: syslog properties: syslog: address: logstash.xxx.jp port: 5514 transport: tcp tls_enabled: true permitted_peer: "*.xxx.jp" ca_cert: | -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- PAS Tile > System Logging
113. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Log Analysis - Kibana Create dashboards for each Biz-Dev-Ops Ops Dev Biz Service status Platform status Apps/System status
114. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Kibana Dashboard Example - Biz Biz # of settlements # of transactions Trend Breakdown Categorization xx,xxx xxx,xxx xx,xxx,xxx,xxx x,xxx,xxx,xxx
115. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Kibana Dashboard Example - Dev # of requests Histogram of latency Dev Other error logs and dashboards etc ... System Error In a merchant High latency in a financial institution
116. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Ops Kibana Dashboard Example - Ops 200 30x 40x 50x Platform Error investigation
117. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Log Alerts Early detection of anomalies through Apps / Platform log alerts. Ops Dev cf_org_name: OrgA log_level: ERROR cf_org_name: OrgA log_level: WARN @message: xxx @message: xxx Set notification destination using cf_org_name, log_level as keys Twilio call for emergency (24 hour support) ElastAlert
118. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ What we achieved ➢ Observability ◦ Metrics ◦ Logging ◦ Tracing
119. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Tracing - Zipkin Visualize communication between microservices Service A Merchant X Financial Institution A API Gateway Receiver A Notification Gateway Time Visualize
120. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Tracing - Zipkin Visualize communication between microservices You can see a bottleneck at a glance across multiple services.
121. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Tracing - Zipkin Visualize communication between microservices Zipkin, Spring Cloud Sleuth <dependency> <groupId>org.springframework.cloud</groupId> <artifactId>spring-cloud-starter-zipkin</artifactId> </dependency> spring: zipkin: base-url: https://my-zipkin.example.com service.name: my-application sender.type: web
122. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Zipkin Tracing Configuration Using Zipkin BOSH Release (OSS) Store traces Ops Dev Biz Search traces Share Elasticsearch with Logging Zipkin BOSH Release https://github.com/making/zipkin-boshrelease Configure instrumentations
123. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Reference Zipkin Tracing Zipkin UI & Kibana # of transactions Duration (Avg) Duratin (Percentile) Sequence Trend
124. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ On-premise Existing app Integrate traces of existing systems (outside PCF) Store traces Existing system team Apps on PCF Store traces Dev Something happens in the existing app? Let me check Visualize communication between new and existing services.
125. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Investigate troubles seamlessly Link to the trace page in Zipkin UI Biz member can easily check
126. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Case study ➢ Case1: Effect on Biz members ➢ Case2: Detect of abnormal trend
127. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Case1: Effect on Business members Specialist of credit card system (NOT developer) Biz # of transactions Duration (Avg) Duratin (Percentile)
128. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Case1: Effect on Business members Specialist of credit card system (NOT developer) Something wrong... It should be shorter... Biz
129. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Specialist of Credit Card System (Not developer) Jump to Zipkin Case1: Effect on Business members This process affects? Biz
130. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Dev Case1: Effect on Business members Biz member can detect the problem. I’ll check now. This transaction seems to be something wrong. Biz
131. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Case2: Detect of abnormal trend
132. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Case2: Detect of abnormal trend With long time period, delayed on specific time slot. 22:00 22:00 22:00 No problem
133. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Case2: Detect of abnormal trend No transaction finished. Transaction might be locked.
134. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ On-premise Existing app Apps on PCF DB access By investigating with Zipkin & Kibana... Case2: Detect of abnormal trend request response delayed... Delays on a specific time slot Re-considered the number of parallels and timeout.
135. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Case2: Detect of abnormal trend Our system was improved by detecting slight abnormality
136. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Observable Platform The environment which provides metrics / logging / tracing “dial tone” Improves apps’ operation efficiency dramatically
137. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Summary
138. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Our journey • Established an engineering team • Improved existing operations • Introduced and built Pivotal Platform and ecosystems. • In-house payment system!
139. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ What we gained by introducing the platform Before After Release Improvement Release Work Manual work One click Release Quality Human error occurs No mistakes Release Time 45 min 5 min Use of Cloud Scaleout operation Manual work One click Container Orchestration - Leave it to the platform Auto-restart Self-made tools Leave it to the platform
140. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ We were able to get the system we wanted! • Fault tolerant, Resilient • Observable • Continuously deliverable Obtained by in-house payment system
141. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Lessons we Learned A platform cannot be built by only relying on outsourced vendors. It’s possible to build and operate a platform by taking ownership in-house. A powerful platform allows a small engineering team to focus on application development.
142. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/