Upgrade to Pro — share decks privately, control downloads, hide ads and more …

A Journey to Developing an In-House Payment System / Spring One Platform 2019

A Journey to Developing an In-House Payment System / Spring One Platform 2019

suzukij

June 25, 2022
Tweet

More Decks by suzukij

Other Decks in Programming

Transcript

  1. A Journey to Developing an In-House Payment System Cloud-Native Development

    with Spring and Pivotal Platform October 7–10, 2019 Austin Convention Center Junya Suzuki (@suzukij), Daichi Kimura, SB Payment Service #springone @s1p
  2. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ About me 2  Part of the SoftBank Group in Japan. Application Engineer Junya Suzuki @suzukij Application engineer focusing on Java and Web systems. Developing new services with the Springframework and improving system operations. Platform Engineer Daichi Kimura Responsible for operational support, promotion and management of Cloud IaaS, PaaS and monitoring platform for payment services.
  3. We issue credit cards "Softbank Card" to consumers. Credit Card

    Issuer Payment Aggregator We provide a comprehensive payment platform that offers various online payment solutions. Credit Card Acquirer We are the only payment aggregator in Japan who accepts and processes transactions made with major brands (VISA/ mastercard/ UnionPay). Softbank customers can also pay for online purchases via their phone bill as Japan’s leading carrier company. Carrier Billing Provider The Largest Payment Service Provider in Japan Our Business
  4. We issue credit cards "Softbank Card" to consumer. Credit Card

    Issuer Payment Aggregator We provide a comprehensive payment platform that offers various online payment solutions. Credit Card Acquirer We are an only payment aggregator in Japan that accepts and processes transactions made with a major brands(VISA/ mastercard/ UnionPay) as an acquirer. We provide consumers to pay for online purchases with their phone bill as Japanese leading carrier company. Carrier Billing Provider The Largest Payment Service Provider in Japan Our Business
  5. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ If you have any questions, we’d be happy to answer them at the end of our presentation
  6. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Agenda • The road to the in-house payment system • Why we chose Pivotal Platform • What we achieved
  7. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ The road to the in-house system
  8. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ As of 2016 ... We left all development of our services to outside vendors. There were zero in-house engineers writing code. The development environment was not ready.
  9. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 2016: Made operations more efficient by ourselves Problem Many mistakes due to manual operations Team
  10. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 2016: Made operations more efficient by ourselves Spring Boot Selenium / Selenide Problem Developed Support Tools Introduced Tools Many mistakes due to manual operations Automate operational tasks Prepared the development environment Team
  11. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 2016: Made operations more efficient by ourselves Problem Many mistakes due to manual operations Team Spring Boot Selenium / Selenide Developed Support Tools Introduced Tools Automate operational tasks 3 Engineers joined! Accelerated KAIZEN tasks by a team
  12. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 2017: ① Improved service monitorings Could not grasp and share the service status quickly. Problem Team
  13. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 2017: ① Improved service monitorings Could not grasp and share the service status quickly. Problem Created dashboards Introduced Tools Elasticsearch Logstash Kibana Team
  14. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 2017: ② Started support for development projects Problem Legacy architectures Expensive develop/release cost Hard to monitor Team
  15. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 2017: ② Started support for development projects Problem Re-architected with Spring Introduced Tools Modern software development  Spring Boot  Spring Cloud Legacy architectures Expensive develop/release cost Hard to monitor Support for proper CI flow Team
  16. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 2017: ② Started support for development projects Problem Re-architected with Spring Introduced Tools Modern software development  Spring Boot  Spring Cloud Legacy architectures Expensive develop/release cost Hard to monitor Team One more engineer joined!
  17. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 2016 - 2017 • Established an engineering team • Developed in-house tools for improved operations • Joined development projects for support 2018 In-house payment system project has finally started
  18. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Merchants Financial Institutions E-Commerce Gaming Education RealEstate Etc E-Books/Movies Unified Payment Service Tickets Provides several payment methods as API to EC sites Credit Card Mobile Carrier Convenience Store Prepaid Card Account Transfer Point Account Integration 当社 Our Company API Development target Online Payment Service
  19. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Merchants Financial Institutions E-Commerce Gaming Education RealEstate Etc E-Books/Movies Unified Payment Service Tickets Provides several payment methods as API to EC sites Credit Card Mobile Carrier Convenience Store Prepaid Card Account Transfer Point Account Integration 当社 Our Company API Development target Online Payment Service Number of adoption 111,742 stores (As of May 2019) Transaction Amount $ 28 billion (As of 2018)
  20. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Merchants Financial Institutions E-Commerce Game Education RealEstate Etc E-Book/Movie Unified Payment Service Ticket Provides several payment methods as API to EC sites Credit Card Mobile Carrier Convenience Store Prepaid Card Account Transfer Point Account Integration 当社 Our Company API Development target Online Payment Service 40+ payment methods supported
  21. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Merchants Financial Institutions E-Commerce Gaming Education RealEstate Etc E-Books/Movies Unified Payment Service Tickets Provides several payment methods as API to EC sites Credit Card Mobile Carrier Convenience Store Prepaid Card Account Transfer Point Account Integration 当社 Our Company API Development target Online Payment Service Located between merchant systems and financial institution systems
  22. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 2018: Developing a In-house system Requirements for the new system • Development and Release with speed • Continuous improvement cycle • Observable, fault-tolerant and resilient Before… Every project was lead by external vendors (A long path from estimation / contract / requirement definition to acceptance)
  23. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 2018: Developing a In-house system Requirements for the new system • Development and Release with speed • Continuous improvement cycle • Observable, fault-tolerant and resilient 今までは… 案件毎に開発ベンダさんのチカラを借りて構築 (見積もり/要件定義から検収まで長い道のり) Outsourcing made it impossible to deliver incrementally and quickly in the agile way.
  24. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Requirements for the new system • Development and Release with speed • Continuous improvement cycle • Observable, fault-tolerant and resilient Speedy delivery and Continuous improvement through in-house development 2018: Developing a In-house system
  25. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Team In-house dev PJ One more engineer joined! 6-person team 2018: In-house Payment System
  26. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 2018: In-house Payment System Introduced Tools Cloud Native Platform based on Pivotal Platform
  27. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Why we chose Pivotal Platform
  28. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ ・Improvements in the release process ・Full utilization of the cloud Requirements for a Platform
  29. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Why not Kubernetes?? As an engineer, k8s is exciting, but ... ・High learning and maintenance costs ・Time required to build a “production-ready” platform ・Insufficient resources to handle k8s OR PaaS
  30. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Why not Kubernetes?? OR PaaS What we needed Needed a PROVEN platform that ・makes developers focus on application development ・enables deploying applications quickly
  31. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ What we finally chose ... PaaS • Great integration with Java/Spring • Support for introducing/operating the platform • cf push experience to deploy apps
  32. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ What we finally chose ... PaaS • Great integration with Java/Spring • Support for introducing/operating the platform • cf push experience to deploy apps Buildpack converts source code to a container image. Developers don’t need to write a Dockerfile. cf push + Buildpack reduces extra works!
  33. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Team structure and responsibility boundary
  34. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Team structure and responsibility boundary Team structure and responsibility boundary Networking Storage Servers Virtualization O/S Middleware Runtime Platform Operators 2 people
  35. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Team structure and responsibility boundary Networking Storage Servers Virtualization O/S Middleware Runtime Data Application Application Developers 4 people Platform Operators 2 people
  36. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Networking Storage Servers Virtualization O/S Middleware Runtime Data Application Focus on building and operating platforms Application Developers 4 people Platform Operators 2 people Team structure and responsibility boundary
  37. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Networking Storage Servers Virtualization O/S Middleware Runtime Data Application Focus on design and implementation of business code Team structure and responsibility boundary Application Developers 4 people Platform Operators 2 people
  38. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Networking Storage Servers Virtualization O/S Middleware Runtime Data Application 12 Factor App The only contract is “12 Factor App”. No vendor lock in. Application Developers 4 people Team structure and responsibility boundary Platform Operators 2 people
  39. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ What we achieved ➢ Platform / Overall Architecture ➢ Architecture ➢ CI/CD Pipelines ➢ Observability
  40. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ syslog+TLS Logstash Elasticsearch Kibana cf push Concourse Prometheus Grafana git push Architecture Overview cf create-service cf bind-service
  41. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ What we achieved ➢ Platform / Overall Architecture ➢ Architecture ➢ CI/CD Pipelines ➢ Observability
  42. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Application Architecture(Merchants ➡ Financial Institutions) API Gateway Service A Service B Service C Merchant X Merchant Y Merchant Z Financial Institution A Financial Institution B Financial Institution C
  43. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Application Architecture(Merchants ➡ Financial Institutions) API Gateway Service A Service B Service C Merchant X Merchant Y Merchant Z Financial Institution A Financial Institution B Financial Institution C EC Sites Shopping、Games、Books、 Tickets、Real Estate and so on
  44. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Application Architecture(Merchants ➡ Financial Institutions) API Gateway Service A Service B Service C Financial Institution A Financial Institution B Financial Institution C New Payment System Merchant X Merchant Y Merchant Z
  45. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Application Architecture(Merchants ➡ Financial Institutions) API Gateway Service A Service B Service C Financial Institution A Financial Institution B Financial Institution C Merchant X Merchant Y Merchant Z Financial Institutions Credit Cards、Convenience Stores、 Mobile Carriers、Prepaid Cards and so on
  46. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Application Architecture(Merchants ➡ Financial Institutions) API Gateway Service A Service B Service C Merchant X Financial Institution A Financial Institution B Financial Institution C Merchant X Merchant Y Merchant Z Each app is deployed on PAS as a microservice
  47. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Application Architecture(Merchants ➡ Financial Institutions) API Gateway Service A Service B Service C Financial Institution A Financial Institution B Financial Institution C Merchant X Merchant Y Merchant Z Each app is implemented with Java and Spring Boot
  48. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Application Architecture(Merchants ➡ Financial Institutions) API Gateway Service A Service B Service C Financial Institution A Financial Institution B Financial Institution C Merchant X Merchant Y Merchant Z API Gateway Routing layer to a Service that implements a business logic per financial institution
  49. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ API Gateway Service A Service B Service C Financial Institution A Financial Institution B Financial Institution C Application Structure( Merchant ➡ Settlement Org) Merchant X Merchant Y Merchant Z
  50. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ API Gateway Service A Service B Service C Merchant X Merchant Y Merchant Z Financial Institution A Financial Institution B Financial Institution C The systems of merchants system and financial institutions are out of our control Application Structure( Merchant ➡ Settlement Org)
  51. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Hystrix API Gateway Service A Service B Service C Merchant A Merchant Y Merchant Z Financial Institution A Financial Institution B Financial Institution C Introduced Hystrix Circuit Breaker for inter-system communications Hystrix Hystrix Hystrix Application Structure( Merchant ➡ Settlement Org)
  52. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ API Gateway Service A Service B Service C Merchant X Merchant Y Merchant Z Financial Institution A Financial Institution B Financial Institution C Without Circuit Breaker, If a system outage happens in financial institution A … Application Structure( Merchant ➡ Settlement Org)
  53. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ API Gateway Service A Service B Service C Merchant X Merchant Y Merchant Z Financial Institution A Financial Institution B Financial Institution C Slow Response, Timeout Application Structure( Merchant ➡ Settlement Org)
  54. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Application Structure( Merchant ➡ Settlement Org) API Gateway Service A Service B Service C Merchant X Merchant Y Merchant Z Financial Institution A Financial Institution B Financial Institution C The failure is propagated to Service A, blocking processes and causing possible thread exhaustion
  55. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ API Gateway Service A Service B Service C Merchant X Merchant Y Merchant Z Financial Institution A Financial Institution B Financial Institution C Failure propagated to API Gateway causing blocked processes, thread depletion Application Structure( Merchant ➡ Settlement Org)
  56. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ API Gateway Service A Service B Service C Merchant X Merchant Y Merchant Z Financial Institution A Financial Institution B Financial Institution C Application Structure( Merchant ➡ Settlement Org) Failure propagated to API Gateway causing blocked processes, thread depletion
  57. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ API Gateway Service A Service B Service C Merchant X Merchant Y Merchant Z Financial Institution A Financial Institution B Financial Institution C B and C are affected by the failure of financial institution A. Application Structure( Merchant ➡ Settlement Org)
  58. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ API Gateway Service A Service B Service C Merchant X Merchant Y Merchant Z Financial Institution A Financial Institution B Financial Institution C Hystrix Hystrix Hystrix Hystrix Application Structure( Merchant ➡ Settlement Org) With Circuit Breaker If a system outage happens in financial institution A …
  59. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ API Gateway Service A Service B Service C Merchant X Merchant Y Merchant Z Financial Institution A Financial Institution B Financial Institution C Hystrix Hystrix Hystrix Circuit Breaker prevents the failure propagation. No worry about the effect to other financial institutions. Application Structure( Merchant ➡ Settlement Org)
  60. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ API Gateway Service A Service B Service C Merchant X Merchant Y Merchant Z Financial Institution A Financial Institution B Financial Institution C Hystrix Hystrix Hystrix Circuit Breaker adds fault tolerance and resiliency to the app Application Structure( Merchant ➡ Settlement Org)
  61. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Notification Gateway Receiver A Receiver B Receiver C Merchant X Merchant Y Merchant Z Financial Institution A Financial Institution B Financial Institution C Hystrix Hystrix Hystrix Hystrix Application Structure( Settlement Org ➡ Merchant)
  62. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Application Structure( Settlement Org ➡ Merchant) Notification Gateway Receiver A Receiver B Receiver C Merchant X Merchant Y Merchant Z Financial Institution A Financial Institution B Financial Institution C Hystrix Hystrix Hystrix Hystrix
  63. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Application Structure( Settlement Org ➡ Merchant) Notification Gateway Receiver A Receiver B Receiver C Merchant X Merchant Y Merchant Z Financial Institution A Financial Institution B Financial Institution C Hystrix Hystrix Hystrix Hystrix Introduced RabbitMQ + Spring Cloud Stream for async processing
  64. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Application Structure( Settlement Org ➡ Merchant) Notification Gateway Receiver A Receiver B Receiver C Merchant X Merchant Y Merchant Z Financial Institution A Financial Institution B Financial Institution C Hystrix Hystrix Hystrix Hystrix
  65. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Application Structure( Settlement Org ➡ Merchant) Notification Gateway Receiver A Receiver B Receiver C Merchant X Merchant Y Merchant Z Financial Institution A Financial Institution B Financial Institution C Hystrix Hystrix Hystrix Hystrix When a failure happens in the merchant system
  66. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Application Structure( Settlement Org ➡ Merchant) Notification Gateway Receiver A Receiver B Receiver C Merchant X Merchant Y Merchant Z Financial Institution A Financial Institution B Financial Institution C Hystrix Hystrix Hystrix Hystrix The message will be diverted to a “Dead Letter Queue” and requeued later
  67. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Application Structure( Settlement Org ➡ Merchant) Notification Gateway Receiver A Receiver B Receiver C Merchant X Merchant Y Merchant Z Financial Institution A Financial Institution B Financial Institution C Hystrix Hystrix Hystrix Even if an outage happens in the merchant, the Circuit Breaker will prevent the failure propagation. Hystrix
  68. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Current state of the production environment
  69. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Current state of the production environment Total Applications(Services) 13 ⤴
  70. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Current state of the production environment Total Instances 47 ⤴
  71. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ What we achieved ➢ Platform / Overall Architecture ➢ Architecture ➢ CI/CD Pipelines ➢ Observability
  72. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ CI / CD Pipeline https://concourse-ci.org/ Concourse
  73. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ https://concourse-ci.org/ Concourse CI / CD Pipeline
  74. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Concourse - Dashboard
  75. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Concourse Pipeline from dev to prod Staging Develop Production Push to a branch triggers the pipeline
  76. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Concourse Pipeline from dev to prod Staging Develop Production
  77. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Unit test for each Java version to detect issues on new versions earlier Concourse Pipeline from dev to prod Staging Develop Production
  78. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Concourse Pipeline from dev to prod Staging Develop Production
  79. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Concourse Pipeline from dev to prod Staging Develop Production
  80. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Develop Cycle by CI Concourse Pipeline from dev to prod Staging Develop Production
  81. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Concourse Pipeline from dev to prod Staging Develop Production
  82. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Merging to master branch triggers deploy to Nexus repository Concourse Pipeline from dev to prod Staging Develop Production
  83. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Production One-click Deploy to production from Nexus Repository using nexus concourse resource Concourse Pipeline from dev to prod
  84. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Testing with API Gateway Service A Service B Service C Financial Institution B Mock Financial Institution B Mock Financial Institution B Mock
  85. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Testing with • Load test • Soak test
  86. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Load Test Problem Cause HTTP transmission processing is low throughput PoolingHttpClientConnectionManager pool setting mistake SQL execution processing low throughput HikariCP DB connection pool setting mistake Unexpected short of Circuit Breaker Hystrix tuning mistakes RabbitMQ publish stopped Memory exhaustion stops queuing messages Throughput 20X Service Down 0
  87. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Soak Test (1 week, 168 hours)
  88. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Load test and E2E test by During development, JMeter tests are automatically executed every day. Notify Slack of report screenshot. Report html cf push
  89. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Load test and E2E test by During development, JMeter tests are automatically executed every day. Notify Slack of report screenshot. Performance testing should be repeated over and over during development to ensure continuous improvement.
  90. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ What we achieved ➢ Observability ◦ Metrics ◦ Logging ◦ Tracing
  91. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ What is overvability? ※Metrics, tracing, and logging  https://peter.bourgon.org/blog/2017/02/21/metrics-tracing-and-logging.html Metrics Logging Tracing Make distributed system easy to measure
  92. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Our Observability choices
  93. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ What we achieved ➢ Observability ◦ Metrics ◦ Logging ◦ Tracing
  94. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Metrics Visualizing Metrics and Alertings Prometheus ・BOSH(VM) Metrics ・CF Metrics ・App Metrics ・etc.
  95. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Metrics - Prometheus  Using Prometheus BOSH Release (OSS) https://github.com/bosh-prometheus/prometheus-boshrelease Prometheus Web UI Grafana Preset Dashboard Preset Alerts
  96. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ rabbitmq Monitor all BOSH VMs No need to configure per instance ⇒ Eliminate extra tasks, missed monitorings { "targets":[ "10.0.1.1", "10.0.2.1", "10.0.3.1", "10.0.1.2", "10.0.2.2", ], "labels":{ "__meta_bosh_deployment":"rabbitmq", "__meta_bosh_job_process_name":"rabbitmq-server" } } Monitor automatically Check deployments Scale out Targets are added automatically BOSH Director
  97. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Org A Monitor all Micrometer Metrics through Promregator No need to configure per app ⇒ Eliminate extra tasks, missed monitorings Collect Metrics per app Collect Metrics for all apps Promregator SpaceA cf push CF Cloud Controller Check apps Devs don’t care Prometheus Prometheus Aggregator for Cloud Foundry https://github.com/promregator/promregator Monitor all Apps Considering migration to Metrics Registrar
  98. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Monitor Apps with Micrometer Collect and Expose Metrics Micrometer <dependency> <groupId>io.micrometer</groupId> <artifactId>micrometer-registry-prometheus</artifactId> </dependency> management: endpoints: web: exposure: include: "*" base-path: /actuator
  99. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Grafana Dashboard Also shows past changes Visualize if each deployment is healthy at this moment Visualize if each VM is healthy at this moment Warning Ops BOSH VM’s status and load Warning
  100. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Grafana Dashboard Ops RabbitMQ status・throughput CF Router status・throughput 30+ dashboards
  101. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Grafana Dashboard Dev Apps Metrics We can always check all metrics in Grafana that we have ever collected from logs ・CPU ・JVM Memory(per area) ・Thread ・GC(frequency、time) ・Classloader
  102. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Grafana Dashboard Dev Apps Metrics Can always check metrics in Grafana we have ever collected from logs ・CPU ・JVM Memory(per area) ・Thread ・GC(frequency、time) ・Classloader Example: GC Metrics
  103. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Alerting Despite lots of dashboards Action is triggered by alerting
  104. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Alertmanager Ops Dev No outsourcing to external monitoring center org_name: OrgA severity: fatal org_name: OrgA severity: /.*/ severity: fatal severity: /.*/ Set route/receiver using org_name, severity as keys Twilio call for emergency (24 hour support) Alert system we developed
  105. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Link to the Grafana dashboard Alerting Example Detect RabbitMQ Dead Letter Queue Call
  106. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Custom Alerts (other than presets) Add custom alerts using BOSH Operations-file ・BOSH VM CPU usage (Preset includes only load average) ・CF app container metrics (CPU・RAM・DISK) ・CF memory usage per Org ・Circuit breaker status ・RabbitMQ DLQ ・RabbitMQ queue depth ・etc. Detect the decrease of the service level before user’s report
  107. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ What we achieved ➢ Observability ◦ Metrics ◦ Logging ◦ Tracing
  108. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Logging Aggregate and Visualize Platform/App Logs using Elastic Stack Log Alerting  ・ElastAlert  ・Praeco
  109. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Logging Aggregate and Visualize Platform/App Logs using Elastic Stack
  110. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Logging - BOSH Elastic Stack Using Elastic Stack BOSH Release (OSS) Search Logs Ops Dev Biz ElastAlert Praeco Alerting Firehose to syslog Syslog + TLS Syslog + TLS Store Logs Apps Platform Elastic Stack BOSH Release https://github.com/bosh-elastic-stack/elastic-stack-bosh-deployment
  111. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Store Logs Firehose-to -syslog Syslog + TLS Apps only need to output logs to stdout Logs are collected by Platform App Logs - aggregated by Firehose-to-syslog Firehose-to -syslog Firehose-to -syslog Loggregator Agent Doppler STD OUT STD OUT Collect Logs Firehose to syslog https://github.com/cloudfoundry-community/firehose-to-syslog Considering migration to Syslog Drain
  112. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Store Logs Syslog + TLS Using BOSH Runtime-Config to apply to all deployments No Need to configure where to forward logs per deployment Platform Logs --- releases: - name: syslog version: "11.4.0" url: "https://bosh.io/xxx?v=11.4.0" sha1: "xxx" addons: - name: syslog exclude: jobs: - name: syslog_forwarder release: syslog jobs: - name: syslog_forwarder release: syslog properties: syslog: address: logstash.xxx.jp port: 5514 transport: tcp tls_enabled: true permitted_peer: "*.xxx.jp" ca_cert: | -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- PAS Tile > System Logging
  113. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Log Analysis - Kibana Create dashboards for each Biz-Dev-Ops Ops Dev Biz Service status Platform status Apps/System status
  114. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Kibana Dashboard Example - Biz Biz # of settlements # of transactions Trend Breakdown Categorization xx,xxx xxx,xxx xx,xxx,xxx,xxx x,xxx,xxx,xxx
  115. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Kibana Dashboard Example - Dev # of requests Histogram of latency Dev Other error logs and dashboards etc ... System Error In a merchant High latency in a financial institution
  116. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Ops Kibana Dashboard Example - Ops 200 30x 40x 50x Platform Error investigation
  117. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Log Alerts Early detection of anomalies through Apps / Platform log alerts. Ops Dev cf_org_name: OrgA log_level: ERROR cf_org_name: OrgA log_level: WARN @message: xxx @message: xxx Set notification destination using cf_org_name, log_level as keys Twilio call for emergency (24 hour support) ElastAlert
  118. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ What we achieved ➢ Observability ◦ Metrics ◦ Logging ◦ Tracing
  119. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Tracing - Zipkin Visualize communication between microservices Service A Merchant X Financial Institution A API Gateway Receiver A Notification Gateway Time Visualize
  120. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Tracing - Zipkin Visualize communication between microservices You can see a bottleneck at a glance across multiple services.
  121. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Tracing - Zipkin Visualize communication between microservices Zipkin, Spring Cloud Sleuth <dependency> <groupId>org.springframework.cloud</groupId> <artifactId>spring-cloud-starter-zipkin</artifactId> </dependency> spring: zipkin: base-url: https://my-zipkin.example.com service.name: my-application sender.type: web
  122. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Zipkin Tracing Configuration Using Zipkin BOSH Release (OSS) Store traces Ops Dev Biz Search traces Share Elasticsearch with Logging Zipkin BOSH Release https://github.com/making/zipkin-boshrelease Configure instrumentations
  123. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Reference Zipkin Tracing Zipkin UI & Kibana # of transactions Duration (Avg) Duratin (Percentile) Sequence Trend
  124. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ On-premise Existing app Integrate traces of existing systems (outside PCF) Store traces Existing system team Apps on PCF Store traces Dev Something happens in the existing app? Let me check Visualize communication between new and existing services.
  125. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Investigate troubles seamlessly Link to the trace page in Zipkin UI Biz member can easily check
  126. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Case study ➢ Case1: Effect on Biz members ➢ Case2: Detect of abnormal trend
  127. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Case1: Effect on Business members Specialist of credit card system (NOT developer) Biz # of transactions Duration (Avg) Duratin (Percentile)
  128. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Case1: Effect on Business members Specialist of credit card system (NOT developer) Something wrong... It should be shorter... Biz
  129. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Specialist of Credit Card System (Not developer) Jump to Zipkin Case1: Effect on Business members This process affects? Biz
  130. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Dev Case1: Effect on Business members Biz member can detect the problem. I’ll check now. This transaction seems to be something wrong. Biz
  131. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Case2: Detect of abnormal trend
  132. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Case2: Detect of abnormal trend With long time period, delayed on specific time slot. 22:00 22:00 22:00 No problem
  133. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Case2: Detect of abnormal trend No transaction finished. Transaction might be locked.
  134. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ On-premise Existing app Apps on PCF DB access By investigating with Zipkin & Kibana... Case2: Detect of abnormal trend request response delayed... Delays on a specific time slot Re-considered the number of parallels and timeout.
  135. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Case2: Detect of abnormal trend Our system was improved by detecting slight abnormality
  136. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Observable Platform The environment which provides metrics / logging / tracing “dial tone” Improves apps’ operation efficiency dramatically
  137. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Summary
  138. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Our journey • Established an engineering team • Improved existing operations • Introduced and built Pivotal Platform and ecosystems. • In-house payment system!
  139. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ What we gained by introducing the platform Before After Release Improvement Release Work Manual work One click Release Quality Human error occurs No mistakes Release Time 45 min 5 min Use of Cloud Scaleout operation Manual work One click Container Orchestration - Leave it to the platform Auto-restart Self-made tools Leave it to the platform
  140. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ We were able to get the system we wanted! • Fault tolerant, Resilient • Observable • Continuously deliverable Obtained by in-house payment system
  141. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Lessons we Learned A platform cannot be built by only relying on outsourced vendors. It’s possible to build and operate a platform by taking ownership in-house. A powerful platform allows a small engineering team to focus on application development.
  142. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software,

    Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/