Serverless Latency

Transcript

1. SERVERLESS LATENCY Serverless Latency is a thing Daniele Frasca Roman Boiko Head of Syndication ProSiebenSat.1 Media SE Specialist Solutions Architect - Serverless AWS © 2023, A mazon Web © 2023, Services, A mazon Inc. or Web its Services, af filiates.Inc. or its af filiates.
2. SERVERLESS LATENCY Daniele Frasca AWS Community Builder @dfrasca80 https://dev.to/ymwjbxxq © 2023, A mazon Web Services, Inc. or its af filiates.
3. SERVERLESS LATENCY Key figures of the ProSiebenSat.1 Group © 2023, A mazon Web Services, Inc. or its af filiates. 5600+ Amazon EC2 instances running each day Over 450 Amazon RDS instances running per day Over 1.5 petabytes transferred via Amazon CloudFront per day Over 340 million AWS Lambda invocations per day Over 1.4 terabytes of data transferred via API Gateway per day Over 12 petabytes of storage stored in Amazon S3
4. SERVERLESS LATENCY Roman Boiko Serverless Specialist SA @romannboiko © 2023, A mazon Web Services, Inc. or its af filiates.
5. SERVERLESS LATENCY Agenda ➢ Latency and concurrency ➢ Architecture examples ➢ Results © 2023, A mazon Web Services, Inc. or its af filiates.
6. SERVERLESS LATENCY We want architecture that supports 10,000 requests per second with 100 ms response time The business © 2023, A mazon Web Services, Inc. or its af filiates.
7. SERVERLESS LATENCY While keeping costs under control, with good performance, and integration with our other services © 2023, A mazon Web Services, Inc. or its af filiates.
8. SERVERLESS LATENCY Can Serverless fit this requirements? © 2023, A mazon Web Services, Inc. or its af filiates.
9. SERVERLESS LATENCY Latency Public Internet Amazon Route 53 + Front door Amazon API Gateway Amazon CloudFront Time AWS Global Accelerator © 2023, A mazon Web Services, Inc. or its af filiates. Application Load Balancer + Computational service AWS Lambda Duration
10. SERVERLESS LATENCY How AWS Lambda concurrency work? © 2023, A mazon Web Services, Inc. or its af filiates.
11. SERVERLESS LATENCY Lambda concurrency Time 1 Initialization © 2023, A mazon Web Services, Inc. or its af filiates. Execution
12. SERVERLESS LATENCY Lambda concurrency Time 1 Initialization © 2023, A mazon Web Services, Inc. or its af filiates. Execution
13. SERVERLESS LATENCY Lambda concurrency Time 1 Initialization Execution Execution environment is blocked/busy for this entire time © 2023, A mazon Web Services, Inc. or its af filiates. 13
14. SERVERLESS LATENCY Lambda concurrency t1 Time 1 Initialization t2 Execution Concurrency == 1 © 2023, A mazon Web Services, Inc. or its af filiates. Concurrency == 1 14
15. SERVERLESS LATENCY Lambda concurrency Time 1 Initialization Execution 2 Initialization Execution 3 Initialization Execution © 2023, A mazon Web Services, Inc. or its af filiates. 15
16. SERVERLESS LATENCY Lambda concurrency t1 Time 1 Initialization t2 Execution 2 Initialization Execution 3 Initialization Execution 3 3 Concurrency © 2023, A mazon Web Services, Inc. or its af filiates. 16
17. SERVERLESS LATENCY Lambda concurrency t3 Time 1 Initialization 2 Initialization Execution 3 Initialization Execution 4 The first environment is now free and warm Execution Initialization 5 Execution Initialization Execution Concurrency 4 © 2023, A mazon Web Services, Inc. or its af filiates.
18. SERVERLESS LATENCY Lambda concurrency t4 Time 1 Initialization Execution 2 Initialization Execution 3 Initialization Execution 4 Initialization 5 Initialization 6 Execution Execution Execution Concurrency 3 © 2023, A mazon Web Services, Inc. or its af filiates.
19. SERVERLESS LATENCY Lambda concurrency t1 Time 1 t2 Initialization t3 t4 6 Execution t5 Execution 2 Initialization Execution 7 Execution 3 Initialization Execution 8 Execution 4 Initialization 5 Execution Initialization t6 9 Execution Execution Concurrency 3 © 2023, A mazon Web Services, Inc. or its af filiates. 4 4 5 4 1
20. SERVERLESS LATENCY Concurrency is a point-in-time measurement © 2023, A mazon Web Services, Inc. or its af filiates.
21. SERVERLESS LATENCY Concurrency is a point-in-time measurement and not the same as requests per second © 2023, A mazon Web Services, Inc. or its af filiates.
22. SERVERLESS LATENCY How to calculate concurrency requirements © 2023, A mazon Web Services, Inc. or its af filiates.
23. SERVERLESS LATENCY Lambda concurrency Lambda concurrency = requests per second (RPS) x duration in seconds Examples • 10,000 RPS x 1,000 ms = 10,000 concurrency • 10,000 RPS x 500 ms = 5,000 concurrency • 10,000 RPS x 100 ms = 1,000 concurrency Long-running functions require more concurrency © 2023, A mazon Web Services, Inc. or its af filiates.
24. SERVERLESS LATENCY T ran sact io n pe r Seco nd Time = 1 second Concurrent requests 1 Init Execution 2 Init Execution 3 Init Execution 4 Init Execution 5 Init Execution 6 Init Execution 7 Init Execution 8 Init Execution 9 Init Execution 10 Init Execution Transaction per second = 10 © 2023, A mazon Web Services, Inc. or its af filiates. If the function duration is 1 second to run 10 invocations happen concurrently Lambda creates 10 runtime environments. https://aws.amazon.com/blogs/compute/understanding-aws-lambda-scaling-and-throughput/
25. SERVERLESS LATENCY T ran sact io n pe r Seco nd Time = 1 second Concurrent requests 1 Init Execution 11 Execution 2 Init Execution 12 Execution 3 Init Execution 13 Execution 4 Init Execution 14 Execution 5 Init Execution 15 Execution 6 Init Execution 16 Execution 7 Init Execution 17 Execution 8 Init Execution 18 Execution 9 Init Execution 19 Execution 10 Init Execution 20 Execution Transaction per second = 20 © 2023, A mazon Web Services, Inc. or its af filiates. If the function duration is 500 ms to run 10 invocations happen concurrently Lambda creates 10 runtime environments. However, transactions per second are now 20. https://aws.amazon.com/blogs/compute/understanding-aws-lambda-scaling-and-throughput/
26. SERVERLESS LATENCY Applied concurrency: Lambda scaling quotas © 2023, A mazon Web Services, Inc. or its af filiates.
27. SERVERLESS LATENCY AWS Lambda function scaling quotas Account concurrency quota Maximum concurrency in a given Region across all functions in an account 1,000 per Region by default This can be increased to customer needs © 2023, A mazon Web Services, Inc. or its af filiates.
28. SERVERLESS LATENCY AWS Lambda function scaling quotas Burst concurrency quota Maximum increase in concurrency for an initial burst of traffic 3,000 in Oregon, N. Virginia, and Ireland 1,000 in Tokyo, Frankfurt, and Ohio 500 in all other Regions After the initial burst, your function concurrency can scale by an additional 500 instances each minute © 2023, A mazon Web Services, Inc. or its af filiates.
29. SERVERLESS LATENCY H ow m u ch do es it sc al e? 1000 3000 300k TPS 250k TPS 500 500 200k TPS 2000 1000 3000 2500 2000 1000 1000 1000 20:00 20:01 100k TPS 0 20:02 © 2023, A mazon Web Services, Inc. or its af filiates. 20:03 20:04 Settings Value Concurrency 3000 Burst 1000 Duration p99 100ms TPS x 1000 100K per sec Concurrency 100
30. SERVERLESS LATENCY La m bd a Op ti m isat io n Time Download the code Bootstrap the runtime Start new execution environment Execute initialization code Execute handler code Small packages Use Lambda Execution Context Avoid Fat Lambda Code Cache assets in the /tmp dir Use parallelism when possible Graviton 2 © 2023, A mazon Web Services, Inc. or its af filiates.
31. SERVERLESS LATENCY Sample architecture © 2023, A mazon Web Services, Inc. or its af filiates.
32. SERVERLESS LATENCY M ov in g t o t he cl ou d Microservice Service Layer Operation 1 Operation 2 Operation 3 Data Layer © 2023, A mazon Web Services, Inc. or its af filiates. Operation 4
33. SERVERLESS LATENCY Meet the TEAM Amazon API Gateway AW S L a mb d a Amazon DynamoDB Amazon S3 Amazon EventB ri dge AW S S te p F u n c ti o ns Amazon SQS Amazon SNS © 2023, A mazon Web Services, Inc. or its af filiates.
34. SERVERLESS LATENCY AWS Cloud AWS Region Amazon EventBridge Amazon DynamoDB Amazon S3 POSTED request Amazon API Gateway AWS Lambda AWS Step Functions Amazon SQS Amazon SNS © 2023, A mazon Web Services, Inc. or its af filiates.
35. SERVERLESS LATENCY Lo ad Test in g 10K 9K RPS 8K 7K 6K • • • • 5K 4K 3K Reach the next step in 1 minute Add 1000 users Hold for 4 minutes Over 15M requests 2K 1K 5 © 2023, A mazon Web Services, Inc. or its af filiates. 25 50 Minutes
36. SERVERLESS LATENCY CloudWatch EMF (Embedded metric format) Eventbridge has no level metrics © 2023, A mazon Web Services, Inc. or its af filiates. No observability Amazon API Gateway Amazon SNS Amazon SQS AWS Lambda Amazon API Gateway Amazon EventBridge Amazon SQS AWS Lambda
37. SERVERLESS LATENCY Ca se 1 Lambda Type Fat Sync © 2023, A mazon Web Services, Inc. or its af filiates. Lambda Execution Context Parallelism
38. SERVERLESS LATENCY AWS Cloud AWS Region 181 ms 25 ms 371 ms 116 ms Amazon EventBridge Amazon DynamoDB Amazon S3 390 ms request 171 ms Amazon API Gateway AWS Lambda 65 ms 88 ms © 2023, A mazon Web Services, Inc. or its af filiates. AWS Step Functions Amazon SQS Amazon SNS
39. SERVERLESS LATENCY H ow m u ch do es it sc al e( th eory) ? 1000 3000 7.8k TPS 6.5k TPS 500 500 5.2k TPS 2000 1000 3000 2500 2000 1000 0 1000 1000 20:00 20:01 20:02 © 2023, A mazon Web Services, Inc. or its af filiates. 2.6k TPS 20:03 20:04 Settings Value Concurrency 3000 Burst 1000 Duration p99 371 ms TPS x 1000 ~2.6K per sec Concurrency 371
40. SERVERLESS LATENCY Sl ow ly b u t st ead y Errors Burst quota 3K Concurrent Limit Achieved around 119K RPM or 1.9K TPS before having errors AWS SDK throwing Rate limit errors p99: 554 ms Concurrency:1088 p99: 363 ms Concurrency:445 p99: 205 ms Concurrency:128 © 2023, A mazon Web Services, Inc. or its af filiates.
41. SERVERLESS LATENCY Ca se 2 Lambda Type Fat Sync Lambda Execution Context Parallelism © 2023, A mazon Web Services, Inc. or its af filiates. @dfrasca80
42. SERVERLESS LATENCY AWS Cloud AWS Region 166 ms 22 ms 351 ms ~5% 121 ms Amazon EventBridge Amazon DynamoDB Amazon S3 364 ms request 156 ms Amazon API Gateway AWS Lambda 63 ms 88 ms © 2023, A mazon Web Services, Inc. or its af filiates. AWS Step Functions Amazon SQS Amazon SNS
43. SERVERLESS LATENCY H ow m u ch do es it sc al e( th eory) ? 1000 3000 8.4k TPS 7k TPS 500 500 5.6k TPS 2000 1000 3000 2500 2000 1000 0 1000 1000 20:00 20:01 20:02 © 2023, A mazon Web Services, Inc. or its af filiates. 2.8k TPS 20:03 20:04 ~7% Settings Value Concurrency 3000 Burst 1000 Duration p99 351 ms TPS x 1000 ~2.8K per sec Concurrency 351
44. SERVERLESS LATENCY Sl ow ly b u t st ead y Burst quota 3K Concurrent Limit Achieved around 119K RPM or 1.9K TPS before having errors Burst Concurrency Throttle error AWS SDK throwing Rate limit errors p99: 721 ms Concurrency:1429 p99: 341 ms Concurrency:470 p99: 345 ms Concurrency:244 © 2023, A mazon Web Services, Inc. or its af filiates.
45. SERVERLESS LATENCY Ca se 3 Lambda Type Fat Sync © 2023, A mazon Web Services, Inc. or its af filiates. Lambda Execution Context Parallelism
46. SERVERLESS LATENCY AWS Cloud AWS Region 175 ms 33 ms 273 ms ~22% 127 ms Amazon EventBridge Amazon DynamoDB Amazon S3 285 ms request 204 ms Amazon API Gateway AWS Lambda 124 ms 94 ms © 2023, A mazon Web Services, Inc. or its af filiates. AWS Step Functions Amazon SQS Amazon SNS
47. SERVERLESS LATENCY H ow m u ch do es it sc al e( th eory) ? 1000 3000 10.8k TPS 9k TPS 500 500 7.2k TPS 2000 1000 3000 2500 2000 1000 0 1000 1000 20:00 20:01 20:02 © 2023, A mazon Web Services, Inc. or its af filiates. 3.6k TPS 20:03 20:04 ~28 % Settings Value Concurrency 3000 Burst 1000 Duration p99 273 ms TPS x 1000 ~3.6K per sec Concurrency 273
48. SERVERLESS LATENCY Sl ow ly b u t st ead y Errors Burst quota 3K Concurrent Limit p99: 843 ms Concurrency:2808 Burst Concurrency Throttle error p99: 1.7s Concurrency:2527 AWS SDK throwing Rate limit errors p99: 578 ms Concurrency:838 p99: 199 ms Concurrency:258 p99: 200 ms Concurrency:128 © 2023, A mazon Web Services, Inc. or its af filiates. Achieved around 130K RPM or 2.1K TPS before having errors
49. SERVERLESS LATENCY Ca se 4 Lambda Type Single Sync © 2023, A mazon Web Services, Inc. or its af filiates. Lambda Execution Context Parallelism
50. SERVERLESS LATENCY AWS Cloud ~92% 21 ms AWS Region request 18 ms 37 ms Amazon API Gateway Amazon EventBridge Amazon SQS AWS Lambda INSERTED 1.9 s 964 ms Amazon EventBridge Amazon SQS Amazon SQS Amazon SQS Amazon SQS Amazon SQS AWS Lambda AWS Lambda AWS Lambda AWS Step Functions AWS Lambda © 2023, A mazon Web Services, Inc. or its af filiates. Amazon DynamoDB AWS Lambda Amazon S3
51. SERVERLESS LATENCY H ow m u ch do es it sc al e( th eory) ? 1000 3000 141k TPS 117k TPS 500 500 95k TPS 2000 1000 3000 2500 2000 1000 0 1000 1000 20:00 20:01 20:02 © 2023, A mazon Web Services, Inc. or its af filiates. 47k TPS 20:03 20:04 ~28% Settings Value Concurrency 3000 Burst 1000 Duration p99 21 ms TPS x 1000 ~47K per sec Concurrency 21
52. SERVERLESS LATENCY Sl ow ly b u t st ead y Errors DynamoDB Latency Burst quota p99: 22 ms Concurrency: 288 300K Errors because the DynamoDB DynamoDB Latency p99: 22 ms Concurrency: 296 DynamoDB Latency p99: 21 ms Concurrency: 176 p99: 161 ms Concurrency: 913 p99: 22 ms Concurrency: 31 © 2023, A mazon Web Services, Inc. or its af filiates.
53. SERVERLESS LATENCY EDA Latency p10 p50 p90 p99 SQS 16,10 ms 18,20 ms 25,10 ms 96,00 ms SQS FIFO 26,60 ms 32,20 ms 40,50 ms 114,00 ms SNS 63,60 ms 77,50 ms 128,00 ms 228,00 ms SNS+SQS 36,80 ms 42,30 ms 57,30 ms 129,00 ms EventBridge 119,00 ms 151,00 ms 215,00 ms 335,00 ms Kinesis 224,00 ms 607,00 ms 984,00 ms 1113,00 ms Kinesis + Enhanced FanOut 47,90 ms 76,00 ms 110,00 ms 159,00 ms Tests run on 05-24-2023 in eu-west-1 region © 2023, A mazon Web Services, Inc. or its af filiates.
54. SERVERLESS LATENCY Ca se 5 - E v ent Br idg e Lambda Type Single Async © 2023, A mazon Web Services, Inc. or its af filiates. Lambda Execution Context Parallelism
55. SERVERLESS LATENCY AWS Cloud AWS Region request ~43% 154 ms 330 ms 36 ms 166 ms Amazon API Gateway Amazon EventBridge IoT core Thing Amazon SQS AWS Lambda 1.9 s 91 ms INSERTED 964 ms Amazon EventBridge Amazon SQS Amazon SQS Amazon SQS Amazon SQS Amazon SQS AWS Lambda AWS Lambda AWS Lambda AWS Step Functions AWS Lambda © 2023, A mazon Web Services, Inc. or its af filiates. Amazon DynamoDB AWS Lambda Amazon S3
56. SERVERLESS LATENCY H ow m u ch do es it sc al e? 1000 3000 19.2k TPS 16k TPS 500 500 12.8k TPS 2000 1000 3000 2500 2000 1000 0 1000 1000 20:00 20:01 20:02 © 2023, A mazon Web Services, Inc. or its af filiates. 6.4k TPS 20:03 20:04 ~77% Settings Value Concurrency 3000 Burst 1000 Duration p99 154 ms TPS x 1000 ~6.4K per sec Concurrency 154
57. SERVERLESS LATENCY Sl ow ly b u t st ead y Results p99: 161 ms Concurrency:123 p99: 157 ms Concurrency:120 p99: 156 ms Concurrency:117 p99: 152 ms Concurrency:134 p99: 149 ms Concurrency:110 p99: 146 ms Concurrency:94 p99: 141 ms Concurrency:73 p99: 139 ms Concurrency:61 p99: 129 ms Concurrency:49 p99: 140 ms Concurrency:23 © 2023, A mazon Web Services, Inc. or its af filiates. Value RPS 10K Concurrency 161 App Type Async Duration p99 < 200 ms Response p99 <1s
58. SERVERLESS LATENCY Ca se 6 - SN S Lambda Type Single Async © 2023, A mazon Web Services, Inc. or its af filiates. Lambda Execution Context Parallelism
59. SERVERLESS LATENCY AWS Cloud AWS Region request 162 ms 123 ms 38 ms 77 ms Amazon API Gateway Amazon SNS IoT core Thing Amazon SQS Amazon DynamoDB AWS Lambda 2s 93 ms INSERTED 441 ms Amazon SNS Amazon SQS Amazon SQS Amazon SQS Amazon SQS Amazon SQS AWS Lambda AWS Lambda AWS Lambda AWS Step Functions AWS Lambda © 2023, A mazon Web Services, Inc. or its af filiates. AWS Lambda Amazon S3
60. SERVERLESS LATENCY H ow m u ch do es it sc al e? 1000 3000 18k TPS 15k TPS 500 500 12k TPS 2000 1000 3000 2500 2000 1000 0 1000 1000 20:00 20:01 20:02 © 2023, A mazon Web Services, Inc. or its af filiates. 6k TPS 20:03 20:04 Settings Value Concurrency 3000 Burst 1000 Duration p99 162 ms TPS x 1000 ~6K per sec Concurrency 162
61. SERVERLESS LATENCY Sl ow ly b u t st ead y Results p99: 161 ms Concurrency:123 p99: 157 ms Concurrency:120 p99: 156 ms Concurrency:117 p99: 152 ms Concurrency:134 p99: 149 ms Concurrency:110 p99: 146 ms Concurrency:94 p99: 141 ms Concurrency:73 p99: 139 ms Concurrency:61 p99: 129 ms Concurrency:49 p99: 140 ms Concurrency:23 © 2023, A mazon Web Services, Inc. or its af filiates. Value RPS 10K Concurrency 161 App Type Async Duration p99 < 200 ms Response p99 < 400 ms
62. SERVERLESS LATENCY Ca se 7 - SQ S Lambda Type Single Async © 2023, A mazon Web Services, Inc. or its af filiates. Lambda Execution Context Parallelism
63. SERVERLESS LATENCY AWS Cloud AWS Region request 166 ms 90 ms 41 ms 92 ms Amazon API Gateway Amazon SQS IoT core Thing AWS Lambda 2.8s 94 ms INSERTED 175 ms Amazon EventBridge Amazon SQS Amazon SQS Amazon SQS Amazon SQS Amazon SQS AWS Lambda AWS Lambda AWS Lambda AWS Step Functions AWS Lambda © 2023, A mazon Web Services, Inc. or its af filiates. Amazon DynamoDB AWS Lambda Amazon S3
64. SERVERLESS LATENCY H ow m u ch do es it sc al e? 1000 3000 18k TPS 12k TPS 500 500 9k TPS 2000 1000 3000 2500 2000 1000 0 1000 1000 20:00 20:01 20:02 © 2023, A mazon Web Services, Inc. or its af filiates. 6k TPS 20:03 20:04 Settings Value Concurrency 3000 Burst 1000 Duration p99 166 ms TPS x 1000 ~6K per sec Concurrency 166
65. SERVERLESS LATENCY Sl ow ly b u t st ead y Results p99: 161 ms Concurrency:161 p99: 176 ms Concurrency:223 p99: 183 ms Concurrency:233 p99: 161 ms Concurrency:229 p99: 158 ms Concurrency:192 p99: 145 ms Concurrency:101 p99: 137 ms Concurrency:99 p99: 129 ms Concurrency:88 p99: 246 ms Concurrency:153 p99: 123 ms Concurrency:38 © 2023, A mazon Web Services, Inc. or its af filiates. Value RPS 10K Concurrency 161 App Type Async Duration p99 < 200 ms Response p99 < 350 ms
66. SERVERLESS LATENCY Synchronous invocations: Best practices Load test •Raise account-level concurrency quota in advance of bursts of demand Set appropriate timeouts •Set reasonable timeouts per Lambda function – do not set to 15 minutes •Configure API response timeout – default is 29 seconds Retry with backoff •Clients should retry, but with exponential backoff to not stampede the system Implement idempotency •Expect and account for retried operations •Application awareness and handling of scenarios is key Dive deeper at https://s12d.com/reInvent2020-Error-Handling © 2023, A mazon Web Services, Inc. or its af filiates.
67. SERVERLESS LATENCY Asynchronous invocations: Best practices Invocation model •Leverage async invocations whenever possible Function configuration • Leverage batching for cost reduction and performance gains Catch errors/failures and return altogether if using a batch size greater than 1 • Configure Lambda on-failure destination to save and review failed messages • • Monitor function error metrics to ensure they remain at or close to 0 Amazon SQS and Lambda best practices: https://s12d.com/SQS-Lambda-Best-Practices © 2023, A mazon Web Services, Inc. or its af filiates.
68. SERVERLESS LATENCY Best pra ct ices - ru n tim e Lambda Type Fat Duration Response Sync ~371 ms < ~400 ms Fat Sync ~331 ms < ~400 ms Fat Sync ~273 ms < ~300 ms Single Sync ~21 ms < ~40 ms EDA - EB Async ~154 ms < ~600 ms EDA - SNS Async ~162 ms < ~400 ms EDA - SQS Async ~166 ms < ~350 ms EDA – SQS (Rust) Async ~60 ms < ~200 ms © 2023, A mazon Web Services, Inc. or its af filiates. Execution Context Parallelism
69. SERVERLESS LATENCY Ser ve rle ss t ake aw ay • Step scaling case • This is the perfect scenario if you are under the burst quota. • Serverless is still better than a similar container app. • Async case • Serverless integration latency is high. • Be aware that each service has different quotas, so it will not just work out of the box. • WebSockets are needed to return responses to clients • Sync case • Only under the burst quota. • Serverless is still better than a similar container app. • Unpredictable traffic • If you are over burst quota, good luck with that ☺ © 2023, A mazon Web Services, Inc. or its af filiates.
70. SERVERLESS LATENCY Thank you! Daniele Frasca Roman Boiko @dfrasca80 @romannboiko Please credit the authors when reusing materials © 2023, A mazon Web © 2023, Services, A mazon Inc. or Web its Services, af filiates.Inc. or its af filiates.