Improve Processing Performance for PayPay Cashback How we 4x the throughput using Akka Stream XIAO Yang Oct, 2021 1

2 - Name: - 肖杨 （XIAO Yang） - From: - Chengdu, Sichuan, China - PayPay: - Since 2019.10 - Tech Lead in CLM team - Interest: - Functional Programming - Distributed system - Akka (Concurrent Toolset for JVM written in Scala) Self Introduction

3 - What is Cashback in PayPay - How Cashback is given in real-time - Performance issue happened - How we improved the performance ToC

4 Cashback in PayPay

5 General Architecture

6 - Akka Stream - Stream processing library - Code as same as Flow chart - Compositional building blocks - Back-Pressure Support - Alpakka Kafka - Kafka connector backed by Akka Stream - Fine tuned Kafka Consumer/Producer for high performance Akka Stream/Alpakka From: Reactive stream processing using Akka streams

7 Akka Stream/Alpakka

8 - User: 15 million - Peak Traffic: 300 TPS Big Campaign in Oct 2019

9 - Cashback can’t show for some transactions after traffic rate is bigger than 250/s - Cashback process can’t support higher traffic - Time required from a transaction is made until cashback granted ⤴ - Some partition stopped - Consumers for topics affect each other What happened

10 - Visualization - Lag of each stage - Throughput of each stage 0. Identify the Bottleneck

11 - Monitor each Stage - Processing Lag - Throughput - External Dependencies - Response time - Throughput - Resource usage - CPU/Memory - DB Performance Dashboard

12 Asynchronous operator in Akka Stream - mapAsync: Accept Future function and Parallelism - Concurrent Processing - Up to n(parallelism) elements - Can use separated thread pool - Not block caller thread - In-Order Processing - Order can be kept when commit Kafka message 1. Optimize the Process

13 Original Configuration: Threads are blocked - High Parallelism - 120 Futures will be created at same time - Default Executor - Java ForkJoinPool with 8 parallelism 1. Optimize the Process Type Parallelism Save Incoming Event DB Write (Blocking) 10 Cashback Evaluation API Call (Non-blocking) 50 Update Event + Save Cashback DB Write (Blocking) 10 SQS Enqueue API Call (Non-blocking) 50

New Configuration: Less blocking - For Blocking Process - Reduce parallelism - Separated fix-sized thread-pool - For Non-blocking Process - Remain enough parallelism - Use default executor (no context switch) 14 1. Optimize the Process Type Latency Parallelism Theoretical Max Throughput 1000/Latency * Parallelism * 30 Save Incoming Event DB Write (Blocking) 20ms~30ms 2 2000 ~ 3000 TPS Cashback Evaluation API Call (Non-blocking) 100ms~300ms 20 2000 ~ 6000 TPS Update Event + Save Cashback DB Write (Blocking) 30ms~40ms 2 1500 ~ 2000 TPS

15 - Fully Handle 700 TPS in performance test - Whole process can finish within 2s - Up to 1200 TPS for Forward Stream - Show cashback result only - 4x+ vs. 250 TPS Mid Result

Can we do better? 16

17 2. Remove Bottleneck Save and update Event in short interval SQS operation took time Update Cashback twice in short interval Read after write by ID

18 2. Remove Bottleneck Write once Update once Ne need to Read Retry in-progress events got lost

- Fully Handle 2000+ TPS in performance test - Whole process can finish within 2s - 2500+ TPS for Forward Stream process - Show cashback result only - 48x+ vs. 250 TPS - And Now - Handle daily traffic from 40 million user base with no lag 19 Result

20 - Akka Stream/Alpakka Kafka helps - Easily control and tune the flow - Take care of back-pressure, in-order process, global error handling - Be careful about blocking operation - Asynchronous operator in Akka Stream again helps a lot - Reduce unnecessary operations before detail tuning - As user traffic continues to grow, there will always be new challenges Summary

Thank You 21