An exploration of black holes: strange failure modes Tammy Bryant Butow, SRE @ Gremlin @tambryantbutow

@tambryantbutow

@tambryantbutow https://www.youtube.com/watch?v=7TIkE-9pZOk

@tambryantbutow Today we will explore strange failure modes that are unexpected and surprising

@tambryantbutow What is a black hole?

A region of a distributed system where gravity is so strong that nothing—no requests or transactions —can escape from it. All IP packets in this region are unable to escape. What is a black hole? @tambryantbutow

Capture IP packets at the transport layer, targeted by supplied port and host arguments. Use existing traffic policing features in the Linux Kernel to drop targeted IP packets. How can we create a black hole? @tambryantbutow

@tambryantbutow https://www.youtube.com/watch?v=P5_Msrdg3Hk

@tambryantbutow What happens when your banking transactions are involved?

https://github.com/GoogleCloudPlatform/bank-of-anthos The architecture of our banking application @tambryantbutow

Does blackholing a critical path service like the Balance Reader result in graceful degradation of the customer experience? https://4503-f37e5de5-39bf-4406-acbc-9c7f2abb0d16.cs-us-east1-wzxb.cloudshell.dev/home @tambryantbutow

https://app.gremlin.com/attacks/new/kubernetes Does blackholing a critical path service like the Balance Reader result in graceful degradation of the customer experience? @tambryantbutow

The balance appears as $--- This could make the user think they have no money in their account @tambryantbutow

The user is still able to make a deposit of $1000 while the Balance Reader service is in a blackhole. @tambryantbutow

The user is unable to send payments. They will see an error that the payment failed due to Balance Reader. @tambryantbutow

The user is unable to send payments. They will see an error that the payment failed due to Balance Reader. @tambryantbutow

@tambryantbutow

Free demo environment to learn about black holes 1. Use this link to install with minikube on google cloud shell: https://ssh.cloud.google.com/cloudshell/editor?show=ide&cloudshell_git_repo=http s://github.com/GoogleCloudPlatform/bank-of-anthos&cloudshell_workspace=.&clo udshell_tutorial=extras/cloudshell/tutorial.md 2. Click minikube → start 3. In Cloud Shell terminal, run kubectl apply -f extras/jwt/jwt-secret.yaml 4. Click <> Cloud Code → Run on Kubernetes 5. To create black holes, create a namespace for gremlin and install gremlin as helm chart https://github.com/gremlin/helm @tambryantbutow

https://github.com/GoogleCloudPlatform/bank-of-anthos Now let’s see how a blackhole impacts our Transaction History service @tambryantbutow

Does blackholing transaction history result in graceful degradation of the customer experience? https://4503-f37e5de5-39bf-4406-acbc-9c7f2abb0d16.cs-us-east1-wzxb.cloudshell.dev/home @tambryantbutow

Does blackholing transaction history result in graceful degradation of the customer experience? https://4503-f37e5de5-39bf-4406-acbc-9c7f2abb0d16.cs-us-east1-wzxb.cloudshell.dev/home @tambryantbutow

@tambryantbutow kubectl scale deployment transactionhistory --replicas=2 What can we do to mitigate against a blackhole? Depending on the service, scaling replicas may work well

@tambryantbutow kubectl get pods Now we have 2 transaction history pods

https://app.gremlin.com/attacks/new/kubernetes Let’s send 50% of transaction history pods into a blackhole @tambryantbutow

https://github.com/GoogleCloudPlatform/bank-of-anthos There will be a very short outage and then the other pod will take over Pod 2: Transaction History Pod 1: Transaction History Deployment Set: Transaction History replicas=2 @tambryantbutow

We are still able to see transaction history and no longer receive error messages. https://4503-f37e5de5-39bf-4406-acbc-9c7f2abb0d16.cs-us-east1-wzxb.cloudshell.dev/home @tambryantbutow

@tambryantbutow What happens when your shopping is involved

A distributed system: e-commerce example @tambryantbutow

@tambryantbutow

@tambryantbutow

@tambryantbutow

@tambryantbutow Does blackholing a non-critical path service like the Ad Service result in graceful degradation of the customer experience? @tambryantbutow

@tambryantbutow

Graceful Degradation Yes, our experiment was successful and our results were what we expected them to be. The blackhole did not negatively impact the customer experience or our overall SLOs. @tambryantbutow

@tambryantbutow What difference sizes of black holes can we experience or create?

@tambryantbutow Micro Stellar Supermassive We can experience and create black holes of all sizes. When creating black holes, start micro and gradually expand the blast radius @tambryantbutow

@tambryantbutow How do black holes impact observability? @tambryantbutow

@tambryantbutow What can we learn from blackhole-related failures? @tambryantbutow

@tambryantbutow How can we use black holes to learn how to make systems more reliable? @tambryantbutow

@tambryantbutow

GoogleCloudPlatform/bank-of-anthos @tambryantbutow

Thank you Get a copy of the O’Reilly ebook Reducing MTTD for High-Severity Incidents gremlin.com/talk/black holes @tambryantbutow

Thank you! @tambryantbutow @tambryantbutow