Distributed Tracing in Serverless Systems

By Ron Yishai - Epsagon

Presented at Functions 2018 / ServerlessDays Toronto

https://functions.events/2018/toronto/ron-yishai/

Transcript

1. Distributed Tracing in Serverless Systems Ron Yishai, Epsagon
2. > whoami Ron Yishai Senior engineer @ Epsagon Cyber security @ IDF Tel Aviv
3. Monitoring/Observability - why do we need it? Track system health Troubleshoot and fix Optimize performance/cost
4. From a monolith…
5. To microservices!
6. Let’s talk Serverless
7. Serverless is great! Pay per use Autoscaling Development velocity
8. Back to Monitoring/Observability Track system health Troubleshoot and fix Optimize performance/cost
9. Slow down! Let’s go one by one…
10. Track system health System == Functions ?
11. The era of APIs We want managed resources Applications become Highly distributed Highly event-driven
 
 Without access to any server!
12. Track system health System > Functions ! Functions APIs Transactions
13. Serverless != Functions theburningmonk.com
14. Troubleshoot and fix Functions are not enough Need: track asynchronous events
16. Transactions
17. Tracing Asynchronous Invocations
18. Tracing Asynchronous Invocations
19. Tracing Asynchronous Invocations
20. Distributed Tracing
21. Distributed Tracing
22. Implementing Distributed Tracing Manual instrumentation •Before/after calls •At the end of each micro service •High maintenance •High potential of errors
23. Serverless apps are very distributed •Complex systems have thousands of functions •What about the developer velocity?
24. Unique challenges to Serverless Timeouts Out of memory Cold starts Retries
25. Can it be done differently in Serverless?
26. Automation can help to keep up with the development speed of Serverless
27. Example
28. dashboard.epsagon.com
29. Back to Monitoring/Observability Track system health Troubleshoot and fix Optimize performance/cost
30. In Serverless Time is Money
31. How much time do you really spend? Our own code API calls Infrastructure overhead
32. Let’s have a quick look 702ms
33. Tracing timeline
34. A real-life example $$$$$$$$$$$$$$$$ How it started…
35. Scanning functions – the easy way Scanning CloudWatch logs using AWS Lambda – every 5 minutes, save to RDS A new Lambda is spawned for every customer’s function (async) Sounds simple and fun! Poll Spawn (async) CloudWatch
36. As time flies… CloudWatch became highly throttled ➔ requests took a very long time ➔ 5K concurrent Lambdas, for 5 minutes, every 5 minutes !!!!
37. Is that all we need? Probably not
38. Observability
39. Track service health
40. Functions Statistics
41. Dynamic Service Map
42. Business Flows Subscribe Transfer Payment
43. Business Flows
44. Thank you! ron@epsagon.com