©PayPay Corporation Multi-Region Architecture at PayPay Tomoki Nishinaka PayPay Corporation Cloud Infrastructure, Infrastructure Technology Department, Payment Product Division, Product Group 

2 ©PayPay Corporation Self-introduction Tomoki Nishinaka Name Company/ department Role PayPay Corporation Infrastructure Technology Department Cloud Infrastructure Tech Lead AWS IAM Identity Center (AWS SSO) Favorite AWS* Service *Amazon Web Services (AWS) 

©PayPay Corporation About PayPay 

4 ©PayPay Corporation About PayPay 55 Million registered PayPay users (as of Feb 2023) *2. Calculated by PayPay based on "Survey on QR Code Payment Usage Trends in Japan in FY2021", published by PAYMENTS JAPAN. Leading QR Code Payment Provider in Japan in both Uptake and Number of Transactions 

5 ©PayPay Corporation About PayPay Leading QR Code Payment Provider in Japan in both Uptake and Number of Transactions 55 Million registered PayPay users (as of Feb 2023) *2. Calculated by PayPay based on "Survey on QR Code Payment Usage Trends in Japan in FY2021", published by PAYMENTS JAPAN. 

6 ©PayPay Corporation About PayPay 1. From App Annie, “Mobile Market Yearbook 2022”. Most Downloaded Apps in Japan 1. PayPay 　Digital Wallet / Payment 2. LINE 　Communications 3. ZOOM Cloud Meetings 　Conference/Business tool 4. Instagram 　Social media 5. Myna Point 　Digital Wallet / Payment 6. YouTube 　Video Sharing / Entertainment 7. Google Maps 　Navigation 8. TikTok 　Social media 9. Amazon Prime Video 　OTT/Entertainment 10 Gmail 　Email No. 1 App Downloads in the country in FY2021 

7 ©PayPay Corporation About PayPay October 2018 Service launch April 2019 April 2020 June 2021 March 2022 More than 1 update every week - that's more than 50 a year 

8 ©PayPay Corporation About PayPay Fina nce O2O Payments Utility Bills/ Government Services Convenient Services Food & drink Supermark ets Convenien ce stores Drugstores 
 Boo ksto res Online Merchants Beauty Retail History Split Bill Themes Analysis Send/Receive T-CARD Bill Payment KYC Hometown tax PCR testing COVID-19 Notifications Myna Points Flea market Sales Revenue Top-up Add Bank PayPay Bank Carrier Billing Credit cards ATM Top-up Pay Later Loan Earn Points Invest PayPay Insurance Food Delivery Coupons Takeout Taxi Table Order Mobile Charge Stamp card Pay Requests Flyer A super app that allows you to meet all your daily needs with just one smartphone Making life more convenient with PayPay 

9 ©PayPay Corporation About PayPay- Product - PayPay Services We are the team that handles all infrastructure PdM/ PMO Designers FE / BE Mobile QA Data ML/AI Product Infrastructure 

©PayPay Corporation 10 Diverse & Multinational Professionals From around 50 countries -World class Engineering Organization 

11 ©PayPay Corporation Flow of this session Overview of PayPay 01 Challenges of multi-region architectures 02 Tips for multi-region architectures 03 Managing multi-region architectures 04 05 Summary 

12 ©PayPay Corporation Flow of this session Overview of PayPay 01 Challenges of multi-region architectures 02 Tips for multi-region architectures 03 Managing multi-region architectures 04 05 Summary 

13 ©PayPay Corporation PayPay Architecture AWS Key Management Service (AWS KMS) Amazon Managed Streaming for Apache Kafka 

14 ©PayPay Corporation PayPay Architecture Running Self-Managed Kubernetes Amazon Managed Streaming for Apache Kafka AWS Key Management Service AWS KMS) 

15 ©PayPay Corporation PayPay Architecture Amazon Managed Streaming for Apache Kafka Running Self-Managed Kafka AWS Key Management Service (AWS KMS) 

16 ©PayPay Corporation PayPay Architecture Running managed database services and TiDB Amazon Managed Streaming for Apache Kafka AWS Key Management Service (AWS KMS) 

17 ©PayPay Corporation PayPay Architecture Building AWS Glue-based Near Real-time Data Infrastructure Amazon Managed Streaming for Apache Kafka AWS Key Management Service AWS KMS) 

18 ©PayPay Corporation PayPay Architecture Building Log Platform in Amazon OpenSearch Service Amazon Managed Streaming for Apache Kafka AWS Key Management Service AWS KMS) 

19 ©PayPay Corporation PayPay's multi-region architecture Application Load Balancer AWS Asia Pacific (Tokyo) Region AWS Asia Pacific (Osaka) Region Replicated Application Load Balancer AWS Key Management Service (AWS KMS) Amazon Elastic Container Registry (Amazon ECR) AWS Key Management Service (AWS KMS) Amazon Elastic Container Registry (Amazon ECR) 

20 ©PayPay Corporation Flow of this session Overview of PayPay 01 Challenges of multi-region architectures 02 Tips for multi-region architectures 03 Managing multi-region architectures 04 05 Summary 

21 ©PayPay Corporation Challenges of multi-region architectures - Interregional synchronization is not as fast as synchronization between AZs and old data may be accessed - APIs that guarantee Strong Consistency cannot be used between regions (Eventual Consistency is used) - Between regions, you must consider write-read data races (prioritize last write/read) Ensuring data integrity and consistency across regions - As resources are created for each region, human labor and financial costs need to be controlled - An unintended setting in one region may cause architecture to run incorrectly Ease of managing resources AZ: Availability Zone 

22 ©PayPay Corporation Flow of this session Overview of PayPay 01 Challenges of multi-region architectures
 02 Tips for multi-region architectures
 03 Managing multi-region architectures
 04 05 Summary 

23 ©PayPay Corporation Tips for multi-region architectures Amazon Aurora Amazon DynamoDB Amazon S3 AWS Key Management Service (AWS KMS) AWS Secrets Manager These tips will focus on AWS-managed services with a particular focus on Amazon S3! 

24 ©PayPay Corporation Amazon Aurora Global Databases using Osaka Region As the replication of global databases takes place in the Storage layer, it is faster than binlog Tokyo Region Writer Reader Storage Osaka Region Reader Reader Storage Replicated Primary Cluster Secondary Cluster 

25 ©PayPay Corporation Amazon Aurora If there is a problem with the Primary Cluster, the Secondary Cluster can be promoted to handle data writes Tokyo Region Writer Reader Storage Osaka Region Reader Reader Storage Primary Cluster Secondary Cluster Replicated 

26 ©PayPay Corporation Amazon Aurora If there is a problem with the Primary Cluster, the Secondary Cluster can be promoted to handle data writes
 Tokyo Region Writer Reader Storage Osaka Region Reader Storage Writer Primary Cluster Secondary Cluster 

27 ©PayPay Corporation Amazon Aurora Use the remove-from-global-cluster API to promote Osaka and complete the process Tokyo Region Writer Reader Storage Osaka Region Reader Storage Primary Cluster Secondary Cluster Writer Execute removal from Osaka side 

28 ©PayPay Corporation Amazon DynamoDB Very simple setup utilizing Global Tables paypay-global-table Tokyo Region Osaka Region paypay-global-table Bidirectional replication 

29 ©PayPay Corporation Amazon DynamoDB The system is being built by dividing up the DBs to be written to by system Strong Consistency Configuration Eventual Consistency Configuration Bidirectional replication Bidirectional replication 

30 ©PayPay Corporation Amazon DynamoDB The system is being built by dividing up the DBs to be written to by system Strong Consistency Configuration Eventual Consistency Configuration Bidirectional replication Eventual consistency cannot be guaranteed between regions - even when using Consistent Read. Bidirectional replication 

31 ©PayPay Corporation Amazon DynamoDB The system is being built by dividing up the DBs to be written to by system Strong Consistency Configuration Eventual Consistency Configuration Bidirectional replication Cost must be checked when replicating huge tables/indexes Bidirectional replication 

32 ©PayPay Corporation AWS Key Management Service (AWS KMS) Support secure system encryption using multi-region keys Multi-region Primary Key Replicated Multi-region Replica Key Tokyo Region Osaka Region 

33 ©PayPay Corporation AWS KMS Single region keys cannot be changed to multi-region keys Single Region Key Tokyo Region Osaka Region Replicated 

34 ©PayPay Corporation AWS Secrets Manager Replicate to Osaka Region along with Secret and run 
 paypay-sec-key paypay-sec-key Tokyo Region Osaka Region Replicated 

35 ©PayPay Corporation AWS Secrets Manager Replicate to Osaka Region along with Secret and run 
 paypay-sec-key Promoted paypay-sec-key Promoting it from replica means it is also possible to change the Secret Value etc. Tokyo Region Osaka Region 

36 ©PayPay Corporation Amazon S3 Bidirectional replication between Tokyo Region <-> Osaka Region Paypay-bucket-apne1 Paypay-bucket-apne3 Osaka to Tokyo Replication Tokyo to Osaka Replication Tokyo Region Osaka Region 

37 ©PayPay Corporation Amazon S3 In the same way as with Amazon Dynamo DB, we are building the system by dividing up the DB to be written to by system Strong Consistency Configuration Eventual Consistency Configuration Osaka to Tokyo Replication Tokyo to Osaka Replication Osaka to Tokyo Replication Tokyo to Osaka Replication 

38 ©PayPay Corporation Amazon S3 + Amazon CloudFront We have enhanced system availability by building Origin Group Osaka to Tokyo Replication Tokyo to Osaka Replication 

39 ©PayPay Corporation How to create an Amazon S3 multi-region architecture Create source Bucket and destination Bucket paypay-source-bucket paypay-destination-bucket Tokyo Region Osaka Region 

40 ©PayPay Corporation paypay-source-bucket paypay-destination-bucket Create IAM Roles to give permission for replication between the source Bucket and destination Bucket Tokyo Region Osaka Region How to create an Amazon S3 multi-region architecture 

41 ©PayPay Corporation Bidirectional replication between Tokyo Region <-> Osaka Region paypay-source-bucket paypay-destination-bucket Tokyo Region Osaka Region Osaka to Tokyo Replication Tokyo to Osaka Replication How to create an Amazon S3 multi-region architecture 

42 ©PayPay Corporation Just creating a replication will not replicate existing Objects paypay-source-bucket Tokyo Region Osaka Region paypay-destination-bucket Osaka to Tokyo Replication Tokyo to Osaka Replication 1 2 How to create an Amazon S3 multi-region architecture 

43 ©PayPay Corporation New Objects will be replicated paypay-source-bucket Tokyo Region Osaka Region paypay-destination-bucket Osaka to Tokyo Replication Tokyo to Osaka Replication 1 2 3 3 How to create an Amazon S3 multi-region architecture 

44 ©PayPay Corporation Ideally, existing Objects will also be replicated paypay-source-bucket Tokyo Region Osaka Region paypay-destination-bucket Osaka to Tokyo Replication Tokyo to Osaka Replication 1 2 3 3 1 2 How to create an Amazon S3 multi-region architecture 

45 ©PayPay Corporation Create replication between Tokyo<->Osaka paypay-source-bucket paypay-destination-bucket Tokyo Region Osaka Region Osaka to Tokyo Replication Tokyo to Osaka Replication How to create an Amazon S3 multi-region architecture 

46 ©PayPay Corporation There are Batch Operations to replicate existing Objects paypay-source-bucket paypay-destination-bucket Tokyo Region Osaka Region Osaka to Tokyo Replication Tokyo to Osaka Replication Batch Operations How to create an Amazon S3 multi-region architecture 

47 ©PayPay Corporation Flow of this session Overview of PayPay 01 Challenges of multi-region architectures 02 Tips for multi-region architectures 03 Managing multi-region architectures 04 05 Summary 

48 ©PayPay Corporation Managing multi-region architectures - Some things, such as Permissions and Policies, should be consistent. - Anyone in the team should be able to create a replication of equal quality. - Make sure that any initial settings required are not left out (making it easier to review). - At the same time, there should be flexibility for the setting of names, tags, and other specific settings such as lifecycles and timeout values. How can we easily manage resources by region? PayPay achieves this using Terraform Module 

49 ©PayPay Corporation Managing multi-region architectures This is a functionality that allows you to put the settings and resources you want to create into a template, without having to define resources individually. What is Terraform Module? Example: I want to create an endpoint to expose externally You can achieve this just by specifying the endpoint names and instance sizes from the Amazon Route53 + Amazon Load Balance + Amazon EC2 configuration. 

50 ©PayPay Corporation Managing multi-region architectures We stopped hardcoding region codes in the application source - We call region codes and resource names from environment variables - Easily manage the release of apps for each region simply by changing the environment variables Check the usage method of each resource and made sure the application side has the same understanding - Check if Strong Consistency is required, of if Eventual Consistency will work, in addition to whether the usage method is compatible with a multi-region architectures
 - As we wanted to control the IAM Roles for each usage method, we also checked this What we have done with the application side apart from the infrastructure 

51 ©PayPay Corporation Points to be improved in a multi-region architecture In the same region, you can specify security groups as the source or the destination. Multi-region security groups If using different regions, this needs to be specified using an CIDR block. AWS IAM Identity Center (AWS SSO) For services released only in the Tokyo region - Only available to use in one region, such as the Tokyo region, and has no replication functionalities etc. - An independent IAM User management platform has been built separately to prepare for unavailability of AWS IAM Identity Center AWS Transfer Family - While this can be used in the Osaka region, as users are independent for each region, this needs to be created for each User. 

52 ©PayPay Corporation Points to be improved in a multi-region architecture Inevitably more expensive than one region Multi-region costs It is not simply about reducing costs, but judging cost-effectiveness and business needs Moving towards greater availability and scalability 

53 ©PayPay Corporation Flow of this session Overview of PayPay 01 Challenges of multi-region architectures 02 Tips for multi-region architectures 03 Managing multi-region architectures 04 05 Summary 

54 ©PayPay Corporation Summary Managed efficiently through use of AWS-managed services Implemented as necessary in consideration of the labor and financial costs It ensures interregional data integrity and consistency We could accurately configure settings and points for each resource PayPay's multi-region architecture - AWS KMS: Multi-region keys - Separately, we can put other resources into templates, and mandate them without missing out any settings 

©PayPay Corporation 55 Diverse & Multinational Professionals From around 50 countries -World class Engineering Organization 

56 ©PayPay Corporation Diverse organizational culture created from talent from over 50 countries A modern developing environment synonymous with a young startup Technical challenges and a scale consistent with a fast-growing business More opportunities for growth as the company evolves Delivering great value to users and making an impact on society Value and challenge for PayPay developers 

57 ©PayPay Corporation Join us! Employment Information: https://about.paypay.ne.jp/career/en/ 

©PayPay Corporation Thank you for listening