Take it to the LIMIT! Lessons Learned After 2(!) Years of Serverless in Production

There are a number of limitation based considerations to make when deploying and maintaining an AWS Lambda based architecture at scale. In this talk, we will discuss a wide variety of constraints we considered or discovered and our workarounds or concessions for handling them. Specifically, we will talk about our experiences with: circumventing API Gateway timeouts and concurrent invocation limits experienced during load testing, contending with dependency artifact max size limits during deployment, minimizing cold start times (plus the subsequent maximum execution duration limits that resulted from it), and handling Glue ETL issues experienced when attempting to point Glue Crawlers to s3 artifacts generated from kinesis streams. In a lot of these scenarios, we did not completely solve the problem but instead came up with a solution that works “well enough”, so we will highlight tradeoffs and pro/cons of the approaches we’ve taken.

Transcript

1. Take it to the LIMIT! Lessons Learned After 2(!) Years of Serverless in Production
2. Principal SWE Oracle Data Cloud, Moat Tech Lead Top Tier Programmatic Ad Exchange Taq Karim @taqkarim
3. Game Plan Here's what we will discuss today. Background What did we build? Why? Architecture What did the thing actually look like? Constraints The fun stuff - what broke? How?
4. A Dollar and a Dream Or, an introduction to the oldest form of advertising
5. O O H M E C C A
6. Programmatic Mobile/Web Ad.
7. OUT OF HOME + PROGRAMMATIC ADS == RIVERSIDE HIGH SCHOOL COMPUTER CLUB
8. 1Find Canvas Any screen with electricity and internet connection will do SALE Wild UP TO 50% OFF!
9. 2Transform Ad We want to plug in existing programmatic resources and transform for OOH screen.
11. 3Add Content! Useful content - like current weather or sports statistics are great.
13. 4??? Wait for industry to catch up and adopt this tech.
15. 1 2 3 2+ billion Adrequests served / day 40,000+ "Adunits" or screens in play Top 5 Demand Side Platforms partnered 5PROFIT! Grow programmatic OOH in leaps and bounds
16. Game Plan Here's what we will discuss today. Background What did we build? Why? Architecture What did the thing actually look like? Constraints The fun stuff - what broke? How?
17. Guts of an Adrequest
19. Architecture AWS Lambda AWS Kinesis AWS Athena AWS Glue
20. The whole shebang SYSTEM ARCHITECTURE
21. Data Collection API ENDPOINTS
22. The whole shebang SYSTEM ARCHITECTURE
23. Data Processing and Reporting ETL JOBS AND AD-HOC BIG DATA QUERYING
24. The whole shebang SYSTEM ARCHITECTURE
25. Logging + Metrics MONITORING / TELEMETRY
26. Game Plan Here's what we will discuss today. Background What did we build? Why? Architecture What did the thing actually look like? Constraints The fun stuff - what broke? How?
27. Everything is always just a little broken Or, constraints at scale and how we managed them
28. Constraints around AWS Limitations Here are the main points we will discuss 1 2 3 Concurrent Invocation Limitations API Gateway Limitations Function Duration Limitations 4 boto/Kinesis Limitations 5 λ Dependency Size Limit
29. Definition How many simultaneous function invocations are available/region. Constraint 1000 requests.
30. What happens? AWS begins to "throttle" functions, failure mode or retry depending on function Examples 429 status_code error, API Gateway Retry X times then fail, SNS, S3
31. Mitigation Strategy Leverage "reserved concurrency" Allow certain functions to "reserve" percentage of available concurrency
32. At least 100 invocations must remain unreserved/account
33. Bulkhead Method Reserved Concurrency as fault tolerance
34. Measurement UnreservedConcurrentExecutions vs Invocations/function metric
35. Divide and conquer API Gateway functions? Mitigate. Kinesis workers? Let fail + retry
36. Pros Prevent runaway invocations from burst activity Cons Account wide limit Cannot access "rest" of concurrency
37. Mitigation Strategy Increase Concurrency Limit Ask AWS to increase concurrency limit via support ticket
38. Pros Easy! For some usecases, legitimate need Cons Not really solving the problem
39. Mitigation Strategy Measure + optimize perf characteristics Using monitoring, find and solve for perf bottlenecks
40. Decrease Function Duration CPU Allocation increases with memory
41. Fine tune/function Cloudwatch Logs publishes a REPORT line
42. Increase Memory for CPU bound tasks This should decrease duration and alleviate concurrency issues Decrease Memory for IO bound tasks This will optimize cost
43. Pros Cost efficiencies Best effort, "lazy" optimizing
44. Constraints around AWS Limitations Here are the main points we will discuss 1 2 3 Concurrent Invocation Limitations API Gateway Limitations Function Duration Limitations 4 boto/Kinesis Limitations 5 λ Dependency Size Limit
45. Definition Lambdas invoked from API gateway must contend with API gateway limitations Constraint(s) 30s max timeout/request 10mb max page size limit
46. Mitigation Strategies Make API Gateway λs dumb Ingest request ➡ validate ➡ drop to queue ➡ respond Apply API design best practices Paginate or write to S3 bucket
47. Pros Ensures faster lambda duration Better API UX Cons Cases where longer computation times are needed (file transfer, etc) May introduce the need for maintaining state
48. 1 2 3 You might not need λ ! More often than not, instrumenting a λ may NOT be the way to go. Potentially long lived API calls Large Payloads Maintaining state
49. Constraints around AWS Limitations Here are the main points we will discuss 1 2 3 Concurrent Invocation Limitations API Gateway Limitations Function Duration Limitations 4 boto/Kinesis Limitations 5 λ Dependency Size Limit
50. Definition A λ function can only run for a max upper limit of time Constraint 15 minute max duration
51. First CRON λ ➡ Query ElasticSearch Then CRON λ ➡ Push Records to Kinesis Finally CRON λ ➡ Update ElasticSearch
52. What if too many records? CRON λ times out CRON λ ➡ Query ElasticSearch ➡ Too many records Failure Mode: retry Continues to retry same invocation ➡ ES runs out of disk (infinite loop!)
53. First CRON λ ➡ Query ElasticSearch TIMEOUT CRON λ ➡ Retries Finally Kinesis ends up with dupes ES not updated correctly
54. Mitigation Strategies Impossible to bump λ timeout Vertical Scaling Horizontal Scaling Instead, tweak around the λ, either by -
55. Vertically Scale Limit / decrease batch size λ ➡ ElasticSearch Fewer records, will complete in time But, "actual work" takes longer This is ok! Technically a "long lived" process
56. Horizontally Scale Currently λ CRON is a single process Provision multiple CRON lambdas Allocate strategy for distributing load For ElasticSearch, use Slices
58. This is a neat solution! ...That just happened to work out since dependency was Elastic Search
59. Generally, two options available Parallelize / scale λ dependencies You might not needs λ!
60. Constraints around AWS Limitations Here are the main points we will discuss 1 2 3 Concurrent Invocation Limitations API Gateway Limitations Function Duration Limitations 4 boto/Kinesis Limitations 5 λ Dependency Size Limit
61. Definition Kinesis asserts max size limits for # of records added/second Constraint 1Mb/s max
62. Mitigation Strategies Leverage KPL Queues up messages and sends in 1Mb batches or, as large as possible For python, leverage the Kinesis Producer Library package
63. At high load, odd behavior Unusually high # of Kinesis send failures Failures expected if another λ is interacting with stream Issue? KPL default setting execute_on_new_thread=True !!! This is culprit KPL tries to send large batches in new threads For failure mode, multiple nearly simultaneous large sends push - all fail
64. Fix is simple, set to `False`
65. Pros Ensures efficient PUTs to Kinesis Cons At or near limit, fine tuning required Generally, AWS tends to "stealth" update libraries - somewhat unreliable
66. Constraints around AWS Limitations Here are the main points we will discuss 1 2 3 Concurrent Invocation Limitations API Gateway Limitations Function Duration Limitations boto/Kinesis Limitations 5 λ Dependency Size Limit
67. Definition There is a max size limit for all λ dependency artifacts Constraint 50Mb, zipped Including layers
68. All λs come with temp storage /tmp folder, max storage size 512Mb Zip 2x ➡ move to /tmp ➡ unzip
69. Pros A lot more space available! Cons Runs up cold start times Horrendous for short lived λs such as API Gateway invocations
70. Mitigation Strategies Remove all unnecessary files Technically, only .pyc files are needed for dependencies to run...
71. Script for removal Walk through dependencies and remove doc, tests, metadata, .py files
72. Pros A fair middle ground Might get away without having to do the double zip hack Cons Only really works for python apps Doesn't solve problem, only delays it
73. Constraints around AWS Limitations Here are the main points we will discuss 1 2 3 Concurrent Invocation Limitations API Gateway Limitations Function Duration Limitations boto/Kinesis Limitations λ Dependency Size Limit
74. Collecting Metrics from λ Functions
75. Cannot leverage StatsD for monitoring λs lack "hostname"s / other unique ID per function
76. λs are not long lived StatsD / DogStatsd are long lived processes Emitting metrics incurs additional duration time Lacking UID, metrics may overwrite each other
77. Mitigation Strategies Use more λs! Cloudwatch Subscription Filter λ Push metrics downstream from CW logs
78. Logging + Metrics MONITORING / TELEMETRY
79. Pros Operate on batches, similar to Kinesis consumer or CRON λ No need for blocking HTTP calls in λ invocation Cons Single threaded - only ONE Subscription Filter allowed / log group
80. Constraints are GOOD! Enforces responsible use of resources Not all use cases suit λs Stateful? Long lived? Not λ. Trade offs, Trade offs, Trade offs! At scale, compromise! Recap Some key ideas to take away from today
81. Have Qs about λs? I'd be happy to expound as a professional courtesy. Architecture I helped design and implement critical serverless subsystems Packaging and Deployment Best Practices I established best practices for CI/CD pipeline integration and automated testing Telemetry and Alerting I worked with key stakeholders to establish metric schemas and alerting practices
82. Social Media, etc Let's connect! Linkedin http://taq.website/in Twitter @taqkarim Slides http://taq.website/pytn
83. Time for Coffee What are your next steps and goals? How much support do you need from investors and what will it get you?