Observability and monitoring of serverless applications

Serverless is fun and easy. But what about monitoring?
Is there a monitoring monster lurking around the corner?

I’ll admit, there are tons of options to tackle the monster. But what features does AWS offer? And more importantly, how can you best use them to make sure you aren’t called out of bed at night?

In this talk, I’ll speak from the heart and share my own experience from using serverless in production for more than a year.

- Should you structure your logs? Of course, but why?
- How do you set up distributed tracing with X-Ray?
- Provide visibility using CloudWatch metrics and dashboards
- Get insight in your log date using CloudWatch Logs Insights

Using the tools and best practices, you’ll tackle the monster no doubt. I’ll do you one better. There won’t be a monster, you’ll have a new best friend!

Transcript

1. Titel Onderwerp Verantwoordelijke afdeing, personen. Datum Serverless: monitoring and observability Nick Van Hoof 2019/10/16 @TheNickVanHoof
2. 2 § Nick Van Hoof § Cloud Developer § Ordina § https://ordina-jworks.github.io/ § https://nickvanhoof.com Whoami
3. 3 Serverless: monitoring and observability Nick Van Hoof 2019/10/16 @TheNickVanHoof
5. 5 Content Serverless: monitoring and observability § Serverless as the new normal? § Challenges - Solutions § Third party tools bit.ly/2MNbHIK ordina-jworks.github.io nickvanhoof.com
6. 6 Content Serverless: monitoring and observability § Serverless as the new normal? § Challenges - Solutions § Third party tools
7. 7 § Scalability Serverless Characteristics
8. 8 Serverless
9. 9 § Scalability Serverless Characteristics
10. 10 § Serverless enables the power of the cloud ØDevOps ØInfrastructure as code ØShort time to production ØBusiness Value Serverless
11. 11 A typical Lambda landscape...
12. 12 A conference application
13. 13 A conference application CREATE_REQUEST_RECEIVED CREATE_REQUEST_VALIDATED SAVED_IN_DATABASE NOTIFICATION_SENT CACHED_TRACE_DATA
14. 14 Content Serverless: monitoring and observability § Serverless as the new normal? § Challenges - Solutions § Third party tools
15. 15 A conference application
16. 16 Serverless application - challenges Serverless: monitoring and observability What if it goes WRONG?
17. 17 Serverless application - challenges Serverless: monitoring and observability Observability becomes a challenge
18. 18 A conference application
19. 19 Serverless application - challenges Serverless: monitoring and observability § Challenge 1: Where did it go wrong? à Distributed system
20. 20 Serverless application - challenges Serverless: monitoring and observability § Challenge 2: Where are the bottlenecks of my system
21. 21 Serverless application - challenges Serverless: monitoring and observability § Challenge 3: Is my system working as expected? Can it handle the load.
22. 22 Challenges à Let’s deal with it à Solutions
23. 23 Serverless application - 3 challenges Serverless: monitoring and observability 1. Finding the error in the distributed system 2. Tackling performance bottlenecks 3. Testing our remote system
24. 24 Challenge 1: Finding the error in a distributed system Serverless: monitoring and observability § Where did it go wrong? à Distributed system
25. 25 Let’s make the bomb explode Finding the error in the distributed system
26. 26 A conference application CREATE_REQUEST_RECEIVED CREATE_REQUEST_VALIDATED SAVED_IN_DATABASE NOTIFICATION_SENT CACHED_TRACE_DATA
27. 27 A conference application CREATE_REQUEST_RECEIVED CREATE_REQUEST_VALIDATED SAVED_IN_DATABASE NOTIFICATION_SENT CACHED_TRACE_DATA
28. 28 What do we need when a bomb goes off? Finding the error in a distributed system NOTIFICATION
29. 29 CloudWatch ALARMS Finding the error in a distributed system
30. 30 CloudWatch ALARMS Finding the error in a distributed system
31. 31 CloudWatch ALARMS – demo time Finding the error in a distributed system
32. 32 We got notified. What do I do now? Finding the error in a distributed system Check the logs
33. 33 A conference application CREATE_REQUEST_RECEIVED CREATE_REQUEST_VALIDATED SAVED_IN_DATABASE NOTIFICATION_SENT CACHED_TRACE_DATA
34. 34 Logging Serverless: monitoring and observability § logging tells a story § logging might tell many stories at once (concurrent invocations) § every part of the story is told in a different place (distributed)
35. 35 A conference application CREATE_REQUEST_RECEIVED CREATE_REQUEST_VALIDATED SAVED_IN_DATABASE NOTIFICATION_SENT CACHED_TRACE_DATA
36. 36 Logging Serverless: monitoring and observability § Challenge: Getting information out of our logs § Solution: Query the logs for information § Challenge: logs coming from different places § Solution: Aggregate the logs § Solution: Correlate the logs
37. 37 Behold.. STRUCTURED LOGGING Serverless: monitoring and observability Structured logging to the rescue
38. 38 Instead of this Serverless: monitoring and observability
39. 39 Structured Logging to the rescue! Serverless: monitoring and observability
40. 40 A conference application traceId traceId traceId
41. 41 Structured logging Serverless: monitoring and observability § Query your logs § Add contextual data § Add traceId
42. 42 CloudWatch Logs Insights Serverless: monitoring and observability § “CloudWatch Logs Insights enables you to explore, analyze, and visualize your logs instantly, allowing you to troubleshoot operational problems with ease.”
43. 43 Structured logging – demo time Serverless: monitoring and observability
44. 44 AWS CloudWatch: Log Insights Serverless: monitoring and observability § Query language to analyze your logs § Picks up fields from our structured logs § Querying over multiple logGroups
45. 45 Challenge 1: where did it go wrong? Finding the error in a distributed system § Notification § Logging - structured • aggregated • correlated • queryable
46. 46 Serverless application - 3 challenges Serverless: monitoring and observability 1. Finding the error in the distributed system 2. Tackling performance bottlenecks 3. Testing our remote system
47. 47 Challenge 2: Tackling performance bottlenecks Serverless: monitoring and observability
48. 48 Solution 2: distributed tracing to identify bottlenecks in the flow Tackling performance bottlenecks XRAY
49. 49 Solution 2: distributed tracing with XRAY Tackling performance bottlenecks § Understand behavior § Analyze performance § Visualizes behavior
50. 50 A conference application
51. 51 Finding the bottleneck Tackling performance bottlenecks bit.ly/2MNbHIK
52. 52 Finding the bottleneck Tackling performance bottlenecks
53. 53 Distributed tracing with XRAY Wrapping it up! Create-lambda-function Save-session-dynamodb-lambda
54. 54 Challenge 2: Find performance bottlenecks Finding the error in a distributed system § Distributed tracing with XRAY § Visualize flow
55. 55 Serverless application - 3 challenges Serverless: monitoring and observability 1. Finding the error in the distributed system 2. Tackling performance bottlenecks 3. Testing our remote system
56. 56 Challenge 3: Testing remote application Serverless: monitoring and observability
57. 57 Smoke Test Serverless: monitoring and observability § Does my distributed system still work as expected?
58. 58 Smoke Test Serverless: monitoring and observability
59. 59 Load Test Serverless: monitoring and observability § Is the application able to handle the load? bit.ly/2MNbHIK
60. 60 Smoke Test Serverless: monitoring and observability § Does my distributed system still work as expected?
61. 61 Visualizing our test results Serverless: monitoring and observability § Add visualizations to dashboard (goto)
62. 62 Challenge 3: Testing the remote system Finding the error in a distributed system § Smoke tests § Performance tests / load tests § Visualize the results
63. 63 Serverless application - 3 challenges Serverless: monitoring and observability 1. Finding the error in the distributed system o structuring logs o aggregating logs o correlating log 2. Tackling performance bottlenecks o distributed tracing with AWS Xray 3. Testing our remote system o smoke testing o load testing o visualizing test result
64. 64 Content Serverless: monitoring and observability § Serverless as the new normal? § Challenges - Solutions § Third party tools
65. 65 Third Party Tools Serverless: monitoring and observability § Lumigo - Monitoring and troubleshooting for serverless application
66. 66 Third Party Tools Serverless: monitoring and observability § Lumigo - Monitoring and troubleshooting for serverless application
67. 67 © 2019 All Rights Reserved. Protego Labs 67
68. 68 © 2019 All Rights Reserved. Protego Labs 68
69. 69 © 2019 All Rights Reserved. Protego Labs 69
70. 70 Third Party Tools Serverless: monitoring and observability § Lumigo - lumigo-cli: npmjs.com/package/lumigo-cli - Best practices blog: lumigo.io/blog/
71. 71 Structured Logging Wrapping it up!
72. 72 Structured Logging Wrapping it up!
73. 73 Distributed tracing with XRAY Wrapping it up!
74. 74 Testing your remote system Wrapping it up!
76. Thank you && Bye Bye bit.ly/2MNbHIK ordina-jworks.github.io nickvanhoof.com @TheNickVanHoof