Observability and resiliency patterns in the cloud - Serverless edition

Public talk presented at the ServerlessDays meetup in Amsterdam.

While observability empowers developers to be in full control over their owned services, resiliency relies on observability to automate failure handling.
In DAZN, using metrics to monitor the known and, even more importantly the unknown is fundamental to build resilient applications. I will show some examples of the steps that can be taken to ensure that your own micro-services are healthy and resilient on different layers of your (serverless) architecture.

Transcript

1. 1 @Sarutule
2. 2 ⊙﹏⊙ @Sarutule
3. 3 (∩╹□╹∩) @Sarutule
4. 4 Observability and resiliency patterns in the cloud Presented by Sara Gerion @Sarutule
5. 5 What is observability? @Sarutule
6. 6 What is resiliency? @Sarutule
7. 7 Let’s take a simple backend service @Sarutule
8. 8 ✅ HTTP endpoint @Sarutule
9. 9 ✅ Running on Node.js - Express.js @Sarutule
10. 10 ✅ Database @Sarutule
11. 11 My simple backend service @Sarutule
12. 12 Translation in AWS-ese @Sarutule
13. 13 Scenario: failures in your runtime application @Sarutule
14. 14 Scenario: failures in your runtime application @Sarutule Database connection issue? Missing environment variable?
15. 15 Logging gives us information related to the errors and operations of an application @Sarutule
16. 16 Logs in JSON format @Sarutule
17. 17 @Sarutule
18. 18 @Sarutule
19. 19 Sample logging enabled on stage & production environments @Sarutule
20. 20 Log correlation-ID’s for traceability @Sarutule
21. 21 @Sarutule
22. 22 Logging: check ✓ @Sarutule
23. 23 Scenario: failures in your underlying infrastructure @Sarutule
24. 24 Metrics give us information related to the state of the underlying infrastructure via discrete values collected during a specific amount of time @Sarutule
25. 25 CloudWatch Metrics @Sarutule
26. 26 USEFUL STRATEGIES Capture missing data ▪ Missing logs ▪ Incorrect number of running resources HTTP requests ▪ Latency of inbound & outbound requests ▪ Http status ▪ Missing http status (error on a TCP / connection layer) Don’t forget to study your data before getting started! Metrics @Sarutule
27. 27 Metrics: check ✓ @Sarutule
28. 28 Alerts tell us over any anomaly happening in our system @Sarutule
29. 29 Missing data ▪ Notify if service unavailable Definition of different alert types ▪ Custom recipients and custom medium based on severity ▪ Better escalation & visibility for stakeholders Study and your data (should this be a metric or the context of a metric?) ▪ Better metrics filtering ▪ Less data pollution Monitor also the stage environment ▪ Get notified of errors before they appear on prod Monitors & alerts @Sarutule
30. 30 Alerts: check ✓ @Sarutule
31. 31 While tackling resiliency, we need to talk about AWS infrastructure @Sarutule
32. 32 @Sarutule
33. 33 Where is my service, physically? @Sarutule
34. 34 Scenario: outage of AZ eu-central-1a @Sarutule
35. 35 Multi AZ replication @Sarutule
36. 36 Load balancing & data replication @Sarutule
37. 37 Can we simplify this? @Sarutule
38. 38 Simplifying with serverless @Sarutule
39. 39 Pros Highly-scalable @Sarutule
40. 40 Pros Resilient @Sarutule
41. 41 Pros Ad-hoc logs and metrics @Sarutule
42. 42 Simplifying with serverless - even when multi-region! @Sarutule
43. 43 ◆ Logging ◆ Metrics ◆ Traceability ◆ Monitors ◆ Alerts Observability patterns @Sarutule
44. 44 ◆ Redundancy of computing units ◆ Replication of data ◆ Load balancing ◆ DNS routing ◆ Delivery at the edge Resiliency patterns @Sarutule
45. 45 ◆ What can go wrong? ◆ How should my system react in case of that specific failure? ◆ Who should be notified, and how? ◆ How can we limit the impact for customers? Useful questions during design phase @Sarutule
46. 46 ⊙﹏⊙ @Sarutule
47. 47 ʘ‿ʘ @Sarutule
48. 48 Thank you @Sarutule