An exploration of black holes: strange failure modes - Craft Conf - Tammy - Gremlin

In this presentation, Tammy will share the many wonders of black holes - not in space - but in computer science. We'll explore what happens when requests never make it out of a blackhole, how this impacts observability, and what we can learn from blackhole-related failures. Tammy will explain how you can use black holes to learn how to make systems more reliable. We'll send requests into black holes on purpose and observe the consequences in detail. This session will be a technical deep dive on strange failure modes that are unexpected and surprising.

Transcript

1. An exploration of black holes: strange failure modes Tammy Bryant Butow, SRE @ Gremlin @tambryantbutow
2. @tambryantbutow
3. @tambryantbutow https://www.youtube.com/watch?v=7TIkE-9pZOk
4. @tambryantbutow Today we will explore strange failure modes that are unexpected and surprising
5. @tambryantbutow What is a black hole?
6. A region of a distributed system where gravity is so strong that nothing—no requests or transactions —can escape from it. All IP packets in this region are unable to escape. What is a black hole? @tambryantbutow
7. Capture IP packets at the transport layer, targeted by supplied port and host arguments. Use existing traffic policing features in the Linux Kernel to drop targeted IP packets. How can we create a black hole? @tambryantbutow
8. @tambryantbutow https://www.youtube.com/watch?v=P5_Msrdg3Hk
9. @tambryantbutow What happens when your banking transactions are involved?
10. https://github.com/GoogleCloudPlatform/bank-of-anthos The architecture of our banking application @tambryantbutow
11. Does blackholing a critical path service like the Balance Reader result in graceful degradation of the customer experience? https://4503-f37e5de5-39bf-4406-acbc-9c7f2abb0d16.cs-us-east1-wzxb.cloudshell.dev/home @tambryantbutow
12. https://app.gremlin.com/attacks/new/kubernetes Does blackholing a critical path service like the Balance Reader result in graceful degradation of the customer experience? @tambryantbutow
13. The balance appears as $--- This could make the user think they have no money in their account @tambryantbutow
14. The user is still able to make a deposit of $1000 while the Balance Reader service is in a blackhole. @tambryantbutow
15. The user is unable to send payments. They will see an error that the payment failed due to Balance Reader. @tambryantbutow
16. The user is unable to send payments. They will see an error that the payment failed due to Balance Reader. @tambryantbutow
17. @tambryantbutow
18. Free demo environment to learn about black holes 1. Use this link to install with minikube on google cloud shell: https://ssh.cloud.google.com/cloudshell/editor?show=ide&cloudshell_git_repo=http s://github.com/GoogleCloudPlatform/bank-of-anthos&cloudshell_workspace=.&clo udshell_tutorial=extras/cloudshell/tutorial.md 2. Click minikube → start 3. In Cloud Shell terminal, run kubectl apply -f extras/jwt/jwt-secret.yaml 4. Click <> Cloud Code → Run on Kubernetes 5. To create black holes, create a namespace for gremlin and install gremlin as helm chart https://github.com/gremlin/helm @tambryantbutow
19. https://github.com/GoogleCloudPlatform/bank-of-anthos Now let’s see how a blackhole impacts our Transaction History service @tambryantbutow
20. Does blackholing transaction history result in graceful degradation of the customer experience? https://4503-f37e5de5-39bf-4406-acbc-9c7f2abb0d16.cs-us-east1-wzxb.cloudshell.dev/home @tambryantbutow
21. Does blackholing transaction history result in graceful degradation of the customer experience? https://4503-f37e5de5-39bf-4406-acbc-9c7f2abb0d16.cs-us-east1-wzxb.cloudshell.dev/home @tambryantbutow
22. @tambryantbutow kubectl scale deployment transactionhistory --replicas=2 What can we do to mitigate against a blackhole? Depending on the service, scaling replicas may work well
23. @tambryantbutow kubectl get pods Now we have 2 transaction history pods
24. https://app.gremlin.com/attacks/new/kubernetes Let’s send 50% of transaction history pods into a blackhole @tambryantbutow
25. https://github.com/GoogleCloudPlatform/bank-of-anthos There will be a very short outage and then the other pod will take over Pod 2: Transaction History Pod 1: Transaction History Deployment Set: Transaction History replicas=2 @tambryantbutow
26. We are still able to see transaction history and no longer receive error messages. https://4503-f37e5de5-39bf-4406-acbc-9c7f2abb0d16.cs-us-east1-wzxb.cloudshell.dev/home @tambryantbutow
27. @tambryantbutow What happens when your shopping is involved
28. A distributed system: e-commerce example @tambryantbutow
29. @tambryantbutow
30. @tambryantbutow
31. @tambryantbutow
32. @tambryantbutow Does blackholing a non-critical path service like the Ad Service result in graceful degradation of the customer experience? @tambryantbutow
33. @tambryantbutow
34. Graceful Degradation Yes, our experiment was successful and our results were what we expected them to be. The blackhole did not negatively impact the customer experience or our overall SLOs. @tambryantbutow
35. @tambryantbutow What difference sizes of black holes can we experience or create?
36. @tambryantbutow Micro Stellar Supermassive We can experience and create black holes of all sizes. When creating black holes, start micro and gradually expand the blast radius @tambryantbutow
37. @tambryantbutow How do black holes impact observability? @tambryantbutow
38. @tambryantbutow What can we learn from blackhole-related failures? @tambryantbutow
39. @tambryantbutow How can we use black holes to learn how to make systems more reliable? @tambryantbutow
40. @tambryantbutow
41. GoogleCloudPlatform/bank-of-anthos @tambryantbutow
42. Thank you Get a copy of the O’Reilly ebook Reducing MTTD for High-Severity Incidents gremlin.com/talk/black holes @tambryantbutow
43. Thank you! @tambryantbutow @tambryantbutow