©PayPay Corporation Infrastructure in PayPay December 12th, 2023 Shinya Aoki Infrastructure Technology Department, PayPay Corporation 

©PayPay Corporation 2 ©PayPay Corporation About me 青木 真也 / Shinya Aoki Name Position Experience Senior Engineering Manager, Infrastructure Technology Dept., Technology Platform & Core Components Div., Product Group, PayPay Corpration I joined Yahoo Japan Corporation (current LY Corporation) in 2010. Initially, I worked as an ID system development engineer and later took on the role of a team leader. Later, I served as the Technical Director within the Payment Finance Division for 3 years, starting in 2016. I was involved with PayPay since its start-up. After being seconded in 2019, I joined PayPay in April 2020. I worked as the Manager for several backend teams and have been in my current position since November 2022. 

©PayPay Corporation About PayPay 

©PayPay Corporation 4 ©PayPay Corporation About PayPay 60 million registered PayPay users （as of Oct. 2023） Calculated by PayPay based on “Survey on QR Code Payment Usage Trends in Japan in FY2021”, published by PAYMENTS JAPAN Leading QR code payment provider in Japan in both usage and number of payments 

©PayPay Corporation 5 ©PayPay Corporation About PayPay 60 million registered PayPay users （as of Oct. 2023） Calculated by PayPay based on “Survey on QR Code Payment Usage Trends in Japan in FY2021”, published by PAYMENTS JAPAN Leading QR code payment provider in Japan in both usage and number of payments 

©PayPay Corporation 6 ©PayPay Corporation About PayPay October 2018 Service launch April 2019 April 2020 June 2021 March 2022 PayPay App is updated about 50 times a year, almost every week 

©PayPay Corporation 7 ©PayPay Corporation About PayPay Finance O2O Payments Utility Bills / Government Services Convenient Food & drink Supermarkets / Convenience stores Drug stores Book stores Online Merchants Beauty Retail History Split Bill Themes Analysis Send/Receive T-CARD Bill payment KYC Hometown tax PCR testing COVID-19 Notifications Myna Points Fleamarket Top-up from Sales Add Bank PayPay Bank Carrier billing Credit cards ATM Top- up Pay Later Loan Earn Points Invest PayPay Insurance Food Delivery Coupons Takeout Taxi Table Order Mobile Charge Stamp card Pay Requests Flyer A super app that supports all daily needs from a smartphone Making life more convenient with PayPay 

©PayPay Corporation 8 ©PayPay Corporation About PayPay – Product – We are the team that handles the Infrastructure for PayPay services PdM PMO Design FE / BE Mobile QA Data ML / AI Product Infra Platform 

©PayPay Corporation Infrastructure in PayPay 

©PayPay Corporation 10 ©PayPay Corporation Infrastructure in PayPay AWS Key Managemen t Service (AWS KMS) Amazon Managed Streaming for Apache Kafka Google Cloud Big Query 

©PayPay Corporation 11 ©PayPay Corporation Infrastructure in PayPay 2018 2019 2020 2021 2022 2023 Region Kubernetes Database Tokyo Region + Osaka Region (Backup only) Osaka Region Primary Cluster Osaka Cluster Secondary Cluster TiDB RDS Aurora 

©PayPay Corporation 12 ©PayPay Corporation Infrastructure in PayPay 2018 2019 2020 2021 2022 2023 Region Kubernetes Database Tokyo Region + Osaka Region (Backup only) Osaka Region Primary Cluster Osaka Cluster Secondary Cluster RDS Aurora TiDB 1. TiDB 2. Multi-Region, Multi-Cluster 

©PayPay Corporation TiDB 

©PayPay Corporation 14 ©PayPay Corporation TiDB Why TiDB ? ● OSS ● Distributed DB ● Worldwide implementation track record in production ● MySQL compatibiilty ● Horizontal scalability ● Strong consistency ● Multiple components for entire DB functionality Background Back in 2019, the following things became concerns for further business growth. ● Upper limit of clusters and tables in Aurora ● Write throughput under high load 

©PayPay Corporation 15 ©PayPay Corporation TiDB TiDB History in PayPay 2018/10/5 2019 2020 2021 2022 2023 Used for Transaction history Also added in payment flow Added in balance management > 100TiB cluster operation Multi-region configuration Horizontal scaling in prod 

©PayPay Corporation 16 ©PayPay Corporation TiDB Good points ● Resolved initial concerns ○ Eliminated capacity concerns through horizontal scaling ○ Improved write performance ● Saved costs in application modifications by high compatibility with MySQL ● Reduced impact upon instance/hardware failure ● Gained knowledge and skills on both existing systems and TiDB 

©PayPay Corporation 17 ©PayPay Corporation TiDB Concerns / improvement points ● Read performance deteriorated slightly ● Physical distribution of Read workloads is not easy from operational and cost perspectives ● Unlimited scaling leads to difficulty in managing costs Good points ● Resolved initial concerns ○ Eliminated capacity concerns through horizontal scaling ○ Improved write performance ● Saved costs in application modifications by high compatibility with MySQL ● Reduced impact upon instance/hardware failure ● Gained knowledge and skills on both existing systems and TiDB 

©PayPay Corporation Multi-Region, Multi-Cluster 

©PayPay Corporation 19 ©PayPay Corporation Multi-Region, Multi-Cluster Situation and Actions ● Previously, a single cluster with multiple availability zones (AZ) was used ● AWS Osaka Region had always been used to store backup data since service launch ● In order to achieve multi-region structure at application level, multi-cluster configuration was mandatory as a new Kubernetes cluster was needed in Osaka Region ● Considering flexibility and availability, we decided to have multiple clusters even in Tokyo Region ● It was necessary to build infrastructure with operational and cost efficiency Background ● AWS Osaka Region became a full-region (March 2021) ● Aim to achieve further availability 

©PayPay Corporation 20 ©PayPay Corporation Application Load Balancer Multi-Region, Multi-Cluster AWS Tokyo Region AWS Key Management Service (AWS KMS) Amazon Elastic Container Registry (Amazon ECR) AWS Osaka Region (Backup) 

©PayPay Corporation 21 ©PayPay Corporation Application Load Balancer Multi-Region, Multi-Cluster Application Load Balancer AWS Osaka Region AWS Tokyo Region Replicated AWS Key Management Service (AWS KMS) Amazon Elastic Container Registry (Amazon ECR) AWS Key Management Service (AWS KMS) Amazon Elastic Container Registry (Amazon ECR) Primary Secondary 

©PayPay Corporation 22 ©PayPay Corporation Multi-Region, Multi-Cluster Cost efficiency ● Scoped functions ● Appropriate sizing Multi-Region ● Standardized configuration management with Terraform module ○ Consistency across regions ○ Same quality no matter who creates resources ○ Flexibility such as Tags ● Automated the control of pod numbers in Tokyo Secondary cluster and Osaka cluster based on Tokyo Primary cluster Multi-Cluster ● Standardized Kubernetes manifest management ● Built Job controlling mechanism under multi-cluster configuration Operational efficiency 

©PayPay Corporation Summary 

©PayPay Corporation 24 ©PayPay Corporation Summary 5 years since PayPay was launched, the infrastructure keeps growing as the service grows The use of TiDB unlocked scale limitations and improved performance To achieve further availability, single cluster/multi-AZ infrastructure was transformed to multi-cluster/multi-region/multi-AZ infrastructure Infrastructure in PayPay Our infrastructure journey never ends... 

©PayPay Corporation 25 Diverse & Multinational Professionals From around 50 countries -World class Engineering Organization- 

©PayPay Corporation 26 ©PayPay Corporation Diverse organizational culture created from talents from over 50 countries A modern development environment being a young startup Technical challenges and a scale consistent with a fast-growing business More opportunities for growth as the company evolves Delivering significant value to users and making on impact on society Values & Challenges for Developers at PayPay 

©PayPay Corporation 27 ©PayPay Corporation Join us! This is our hiring page: https://about.paypay.ne.jp/career/ 

©PayPay Corporation Thank you