With Great Scalability Comes Great Responsibility

This is a story of how I took down one of our vendor’s services with an innocent serverless application. I wanted to retrieve data from one of our monitoring platforms to analyze SPS Commerce’s software performance. Initially, I wrote a script to collect the data using python multiprocessing. To gather this data in a more scalable, fast, and efficient way, I decided to pivot to a serverless architecture. Unfortunately, my solution ended up spawning requests faster than the REST API could handle. In this talk, we will cover the contextual pros and cons of a number of architectural patterns given real world scalability constraints; from orchestrating Lambdas with AWS step functions to multiprocessing with S3 triggers to rate limiting with queues like SQS.

Transcript

1. WITH GREAT SCALABILITY COMES GREAT RESPONSIBILITY @BigDana Dana Engebretson
2. @BigDana
3. Queen of the Amazons @BigDana
4. Amazon Warriors* *Our services in AWS @BigDana
5. @BigDana
6. @BigDana Dana Engebretson Performance Engineer SPS Commerce
7. @BigDana Attempt 0: Python Multiprocessing on an EC2 Instance Attempt 0.5: Python Multiprocessing on a Spot Instance Attempt 1: Lambdas Orchestrated by Step Functions Attempt 2: Using Event Triggers with S3 Attempt 3: Rate limiting with Queues and Cloudwatch triggered Lambdas Building a Data Pipeline
8. @BigDana Attempt 0: Python Multiprocessing on an EC2 Instance Attempt 0.5: Python Multiprocessing on a Spot Instance Attempt 1: Lambdas Orchestrated by Step Functions Attempt 2: Using Event Triggers with S3 Attempt 3: Rate limiting with Queues and Cloudwatch triggered Lambdas Building a Data Pipeline
9. @BigDana Attempt 0: Python Multiprocessing on an EC2 Instance Attempt 0.5: Python Multiprocessing on a Spot Instance Attempt 1: Lambdas Orchestrated by Step Functions Attempt 2: Using Event Triggers with S3 Attempt 3: Rate limiting with Queues and Cloudwatch triggered Lambdas Building a Data Pipeline
10. @BigDana Attempt 0: Python Multiprocessing on an EC2 Instance Attempt 0.5: Python Multiprocessing on a Spot Instance Attempt 1: Lambdas Orchestrated by Step Functions Attempt 2: Using Event Triggers with S3 Attempt 3: Rate limiting with Queues and Cloudwatch triggered Lambdas Building a Data Pipeline
11. @BigDana Attempt 0: Python Multiprocessing on an EC2 Instance Attempt 0.5: Python Multiprocessing on a Spot Instance Attempt 1: Lambdas Orchestrated by Step Functions Attempt 2: Using Event Triggers with S3 Attempt 3: Rate limiting with Queues and Cloudwatch triggered Lambdas Building a Data Pipeline
12. @BigDana Attempt 0: Python Multiprocessing on an EC2 Instance Attempt 0.5: Python Multiprocessing on a Spot Instance Attempt 1: Lambdas Orchestrated by Step Functions Attempt 2: Using Event Triggers with S3 Attempt 3: Rate limiting with Queues and Cloudwatch triggered Lambdas Building a Data Pipeline
13. @BigDana The Oracle* *one of our monitoring vendors
14. @BigDana Hermes: the messenger God* *Our Vendor’s Api The Oracle* *one of our monitoring vendors
15. @BigDana
16. @BigDana
17. @BigDana FETCH ME ALL MY DATA FOR ALL MY AMAZON WARRIORS PLEASE! The Oracle* *one of our monitoring vendors Hermes: the messenger God* *Our Vendor’s Api
18. Hermes: the messenger God* *Our Vendor’s Api @BigDana The Oracle* *one of our monitoring vendors
19. @BigDana The Oracle* *one of our monitoring vendors Hermes: the messenger God* *Our Vendor’s Api
20. @BigDana HERE IS ALL YOUR DATA! The Oracle* *one of our monitoring vendors Hermes: the messenger God* *Our Vendor’s Api
21. @BigDana
22. @BigDana Hermes: the messenger God* *Our Vendor’s Api @BigDana FETCH ME ALL MY AMAZON WARRIOR TROOPS* PLEASE! *grouped by micro-service The Oracle* *one of our monitoring vendors
23. @BigDana The Oracle* *one of our monitoring vendors Hermes: the messenger God* *Our Vendor’s Api @BigDana
24. @BigDana The Oracle* *one of our monitoring vendors @BigDana Hermes: the messenger God* *Our Vendor’s Api @BigDana
25. TROUP 2 @BigDana @BigDana Hermes: the messenger God* *Our Vendor’s Api @BigDana @BigDana The Oracle* *one of our monitoring vendors TROUP 2 troop 3 TROUP 1 HERE ARE YOUR AMAZON WARRIOR TROOPS!
26. @BigDana @BigDana Hermes: the messenger God* *Our Vendor’s Api @BigDana @BigDana @BigDana The Oracle* *one of our monitoring vendors troop 2 troop 3 troop 1
27. @BigDana z FOR TROOP 1, FETCH ME MY AMAZON WARRIORS PLEASE! The Oracle* *one of our monitoring vendors Hermes: the messenger God* *Our Vendor’s Api troop 2 troop 3 troop 1
28. @BigDana The Oracle* *one of our monitoring vendors @BigDana Hermes: the messenger God* *Our Vendor’s Api @BigDana troop 2 troop 3 troop 1
29. @BigDana The Oracle* *one of our monitoring vendors Hermes: the messenger God* *Our Vendor’s Api troop 2 troop 3 troop 1
30. @BigDana The Oracle* *one of our monitoring vendors Hermes: the messenger God* *Our Vendor’s Api TROUP 1 amazon warrior 1 amazon warrior 2 TROUP 2 TROUP 3 TROUP 1 troop 2 troop 3 troop 1 HERE ARE YOUR AMAZON WARRIORS FOR TROOP 1!
31. @BigDana amazon warrior 1 amazon warrior 2 The Oracle* *one of our monitoring vendors Hermes: the messenger God* *Our Vendor’s Api troop 2 troop 3 troop 1
32. amazon warrior 1 amazon warrior 2 Hermes: the messenger God* *Our Vendor’s Api @BigDana FOR AMAZON WARRIOR 1, FETCH ME MY DATA GROUPS PLEASE! The Oracle* *one of our monitoring vendors troop 2 troop 3 troop 1
33. @BigDana The Oracle* *one of our monitoring vendors Hermes: the messenger God* *Our Vendor’s Api amazon warrior 1 amazon warrior 2 troop 2 troop 3 troop 1
34. @BigDana The Oracle* *one of our monitoring vendors amazon warrior 1 amazon warrior 2 Hermes: the messenger God* *Our Vendor’s Api troop 2 troop 3 troop 1
35. TROUP 2 TROUP 1 amazon warrior 2 TROUP 2 Hermes: the messenger God* *Our Vendor’s Api @BigDana data group 1 data group 2 The Oracle* *one of our monitoring vendors amazon warrior 1 data group 3 troop 2 troop 3 troop 1 HERE ARE YOUR DATA GROUPS FOR AMAZON WARRIOR 1!
36. @BigDana The Oracle* *one of our monitoring vendors amazon warrior 2 Hermes: the messenger God* *Our Vendor’s Api @BigDana data group 1 data group 2 amazon warrior 1 data group 3 troop 2 troop 3 troop 1
37. amazon warrior 2 Hermes: the messenger God* *Our Vendor’s Api @BigDana data group 1 data group 2 amazon warrior 1 data group 3 @BigDana FOR AMAZON WARRIOR 1, AND DATA GROUP 1, FETCH ME THE DATA PLEASE! The Oracle* *one of our monitoring vendors troop 2 troop 3 troop 1
38. @BigDana The Oracle* *one of our monitoring vendors @BigDana Hermes: the messenger God* *Our Vendor’s Api amazon warrior 2 data group 1 data group 2 data group 3 amazon warrior 1 troop 2 troop 3 troop 1
39. @BigDana The Oracle* *one of our monitoring vendors amazon warrior 2 Hermes: the messenger God* *Our Vendor’s Api @BigDana data group 1 data group 2 amazon warrior 1 data group 3 troop 2 troop 3 troop 1
40. amazon warrior 2 Hermes: the messenger God* *Our Vendor’s Api data group 2 amazon warrior 1 data group 3 @BigDana The Oracle* *one of our monitoring vendors data data group 1 TROUP 1 troop 2 troop 3 troop 1 HERE IS YOUR DATA FOR AMAZON WARRIOR 1 AND DATA GROUP 1!
41. amazon warrior 2 Hermes: the messenger God* *Our Vendor’s Api data group 2 amazon warrior 1 data group 3 @BigDana The Oracle* *one of our monitoring vendors data data group 1 troop 2 troop 3 troop 1
42. @BigDana 35,000-65,000 API CALLS!
43. HOW SHOULD I WRITE THE CODE TO COMMUNICATE WITH HERMES? @BigDana
44. @BigDana Attempt 0: Python Multiprocessing on an EC2 Instance Attempt 0.5: Python Multiprocessing on a Spot Instance Attempt 1: Lambdas Orchestrated by Step Functions Attempt 2: Using Event Triggers with S3 Attempt 3: Rate limiting with Queues and Cloudwatch triggered Lambdas Building a Data Pipeline
45. Initial Solution: Python Multiprocessing on an EC2 instance - continuously running Easy for me to implement not cheap horizontal scaling seemed overkill on-going maintenance and management @BigDana
46. Initial Solution: Python Multiprocessing on an EC2 instance - intermittently running Easy for me to implement cheaper still not cheap enough horizontal scaling seemed overkill on-going maintenance and management @BigDana
47. @BigDana Attempt 0: Python Multiprocessing on an EC2 Instance Attempt 0.5: Python Multiprocessing on a Spot Instance Attempt 1: Lambdas Orchestrated by Step Functions Attempt 2: Using Event Triggers with S3 Attempt 3: Rate limiting with Queues and Cloudwatch triggered Lambdas Building a Data Pipeline
48. @BigDana Attempt 0: Python Multiprocessing on an EC2 Instance Attempt 0.5: Python Multiprocessing on a Spot Instance Attempt 1: Lambdas Orchestrated by Step Functions Attempt 2: Using Event Triggers with S3 Attempt 3: Rate limiting with Queues and Cloudwatch triggered Lambdas Building a Data Pipeline
49. Revision: significantly cheaper You need to configure how to restart where the previous process left off @BigDana Python Multiprocessing on an EC2 instance a Spot instance
50. Should I try Serverless? @BigDana
51. Owning your own server : Owning your own car @BigDana
52. The Cloud: Leasing a car Owning your own car: Owning your own server @BigDana
53. The Cloud : Leasing a car Owning your own server : Owning your own car Spot Instances: Renting a car @BigDana
54. The Cloud: Leasing a car Owning your own server: Owning your own car Spot Instances: Renting a car Lambdas: car2go @BigDana
55. @BigDana Attempt 0: Python Multiprocessing on an EC2 Instance Attempt 0.5: Python Multiprocessing on a Spot Instance Attempt 1: Lambdas Orchestrated by Step Functions Attempt 2: Using Event Triggers with S3 Attempt 3: Rate limiting with Queues and Cloudwatch triggered Lambdas Building a Data Pipeline
56. @BigDana Attempt 0: Python Multiprocessing on an EC2 Instance Attempt 0.5: Python Multiprocessing on a Spot Instance Attempt 1: Lambdas Orchestrated by Step Functions Attempt 2: Using Event Triggers with S3 Attempt 3: Rate limiting with Queues and Cloudwatch triggered Lambdas Building a Data Pipeline
57. Lambda: a Function as a Service @BigDana
58. Lambda: a Function as a Service Compute: proportional to memory Time: max of 5 minutes Memory: max of 1536 MB @BigDana
59. Lambda: Function as a Service Compute Time Memory Dependency Zip File size 250 MB uncompressed code/dependencies @BigDana
60. @BigDana
61. @BigDana
62. @BigDana
63. @BigDana
64. @BigDana
65. @BigDana
66. @BigDana
67. Lambda: Function as a Service Compute Time Memory Dependency Zip File size 250 MB uncompressed code/dependencies @BigDana fastparquet didn’t fit
68. Lambda: Function as a Service Compute Time Memory Dependency Zip File size @BigDana Ephemeral Disk Capacity: max of 512 MB
69. Lambda: Function as a Service Compute Time Memory Ephemeral Disk Capacity Dependency Zip File size @BigDana concurrent executions: default max of 1000 per account
70. Lambda: Function as a Service Compute Time Memory Ephemeral Disk Capacity Dependency Zip File size # of file descriptors # of processes and threads Invoke request body payload size @BigDana concurrent executions
71. Lambda: Function as a Service Compute Time Memory Ephemeral Disk Capacity Dependency Zip File size # of file descriptors # of processes and threads Invoke request body payload size Linux @BigDana concurrent executions
72. @BigDana
73. Step Functions start end @BigDana
74. Choice State start end choice @BigDana
75. Parallel Processing Step Functions can do start end @BigDana
76. start end start end … Dynamic Fan Out Parallel Processing Parallel Processing Step Functions can do vs @BigDana
77. @BigDana
78. start … Dynamic Fan Out Parallel Processing @BigDana start Defined Fan Out Parallel Processing vs … …
79. start send to an EC2 instance @BigDana start Defined Fan Out Parallel Processing or EC2 … …
80. Step Functions easy to implement workflow management for lambdas and ec2 instances built-in back off policy for retrying lambdas doesn’t yet natively support a dynamic one-to-many fan out architecture of lambdas @BigDana
81. @BigDana
82. @BigDana Attempt 0: Python Multiprocessing on an EC2 Instance Attempt 0.5: Python Multiprocessing on a Spot Instance Attempt 1: Lambdas Orchestrated by Step Functions Attempt 2: Using Event Triggers with S3 Attempt 3: Rate limiting with Queues and Cloudwatch triggered Lambdas Building a Data Pipeline
83. @BigDana Attempt 0: Python Multiprocessing on an EC2 Instance Attempt 0.5: Python Multiprocessing on a Spot Instance Attempt 1: Lambdas Orchestrated by Step Functions Attempt 2: Using Event Triggers with S3 Attempt 3: Rate limiting with Queues and Cloudwatch triggered Lambdas Building a Data Pipeline
84. S3 Bucket (Simple Storage Service) @BigDana
85. Using Event Triggers troops/ amazon_warriors/ data_groups/ data/ S3 Bucket (Simple Storage Service) @BigDana
86. Using Event Triggers start S3 Bucket (Simple Storage Service) @BigDana troops/ amazon_warriors/ data_groups/ data/
87. Using Event Triggers start S3 Bucket (Simple Storage Service) @BigDana troops/ amazon_warriors/ data_groups/ data/
88. Using Event Triggers start S3 Bucket (Simple Storage Service) Circular dependency @BigDana troops/ amazon_warriors/ data_groups/ data/
89. Using Event Triggers start S3 Bucket (Simple Storage Service) @BigDana troops/ amazon_warriors/ data_groups/ data/
90. Using Event Triggers start S3 Bucket (Simple Storage Service) @BigDana troops/ amazon_warriors/ data_groups/ data/
91. @BigDana
92. Using Event Triggers start S3 Bucket (Simple Storage Service) @BigDana troops/ amazon_warriors/ data_groups/ data/
93. Using Event Triggers start S3 Bucket (Simple Storage Service) @BigDana troops/ amazon_warriors/ data_groups/ data/
94. Using Event Triggers start S3 Bucket (Simple Storage Service) @BigDana troops/ amazon_warriors/ data_groups/ data/
95. @BigDana Too Fast
96. s3 is highly scalable and durable Nested event triggers creates super fast multiprocessing You can’t control the rate the lambdas get spawned It will only retry a lambda 2 times (can extend this manually) You can lose in-flight tasks if there is a system outage Using Event Triggers with S3 @BigDana
97. @BigDana
98. @BigDana You can’t run into a rate limit if you never scale.
99. @BigDana Attempt 0: Python Multiprocessing on an EC2 Instance Attempt 0.5: Python Multiprocessing on a Spot Instance Attempt 1: Lambdas Orchestrated by Step Functions Attempt 2: Using Event Triggers with S3 Attempt 3: Rate limiting with Queues and Cloudwatch triggered Lambdas Building a Data Pipeline
100. @BigDana Attempt 0: Python Multiprocessing on an EC2 Instance Attempt 0.5: Python Multiprocessing on a Spot Instance Attempt 1: Lambdas Orchestrated by Step Functions Attempt 2: Using Event Triggers with S3 Attempt 3: Rate limiting with Queues and Cloudwatch triggered Lambdas Building a Data Pipeline
101. SQS Queues @BigDana (Simple Queue Service)
102. @BigDana
103. @BigDana
104. @BigDana
105. @BigDana
106. @BigDana
107. @BigDana
108. @BigDana
109. @BigDana
110. @BigDana
111. @BigDana
112. @BigDana
113. @BigDana
114. @BigDana
115. @BigDana
116. Cloudwatch Rules @BigDana
117. Troops Amazon Warriors Data Groups Rate limiting with Queues and Cloudwatch triggered Lambdas raw_data/ processed/ data/ @BigDana
118. Rate limiting with Queues and Cloudwatch triggered Lambdas raw_data/ processed/ raw_data/ processed/ @BigDana Data Groups Troops Amazon Warriors
119. @BigDana Queue Depth Over Time 0 7500 15000 22500 30000 12 AM 2 AM 4 AM 6 AM 8 AM 10 AM 12 PM 2 PM 4 PM 6 PM 8 PM 10 PM AMAZON WARRIOR TROOPS AMAZON WARRIORS DATA GROUPS
120. raw_data/ processed/ Rate limiting with SQS Queues and Cloudwatch triggered Lambdas Dead Letter Queues @BigDana Data Groups Troops Amazon Warriors
121. @BigDana Queue Depth Over Time 0 7500 15000 22500 30000 12 AM 2 AM 4 AM 6 AM 8 AM 10 AM 12 PM 2 PM 4 PM 6 PM 8 PM 10 PM AMAZON WARRIOR TROOPS AMAZON WARRIORS DATA GROUPS
122. @BigDana Queue Depth Over Time 0 7500 15000 22500 30000 12 AM 2 AM 4 AM 6 AM 8 AM 10 AM 12 PM 2 PM 4 PM 6 PM 8 PM 10 PM AMAZON WARRIOR TROOPS AMAZON WARRIORS DATA GROUPS
123. Rate limiting with Queues and Cloudwatch triggered Lambdas @BigDana
124. Rate limiting with Queues and Cloudwatch triggered Lambdas @BigDana
125. Rate limiting with Queues and Cloudwatch triggered Lambdas @BigDana
126. Rate limiting with Queues and Cloudwatch triggered Lambdas @BigDana
127. highly scalable and resilient can store failed tasks indefinitely gives you control to rate limit the processing of the tasks in the queue a small task can wait in line on a queue that already has many tasks SQS Queues: @BigDana
128. allows for quite specific schedules fastest schedule is every 1 minute doesn’t scale responsively to your queue depth Lambdas triggered by Cloudwatch Rules @BigDana
129. @BigDana IT TAKES REAL CHARACTER TO ADMIT ONES FAILURES — AND NOT A LITTLE WISDOM TO TAKE YOUR PROFITS FROM DEFEAT
130. @BigDana Lambdas are not yet suitable for quick tasks if they require large dependencies Step Functions does not yet provide dynamic fan out parallel processing Event triggers don’t yet allow you to rate limit We can’t yet trigger a Lambda based off an SQS Queue depth Takeaways
131. @BigDana Expect Limits Expect to Exceed those limits Expect you will have to handle failures Lessons Learned
132. @BigDana Invest in training new hires Decouple the learning process of your build deploy pipeline and building serverless architecture Reflections for Managers
133. @BigDana Victoria Pierce @thesecondshade June 5th 2017: Joined SPS Commerce as a Site Reliability Engineer Intern July 12th 2017: Deployed her first lambda As of November 15th 2017: Maintains 6 serverless applications
134. @BigDana Thank you! Dana Engebretson Performance Engineer SPS Commerce Questions?