When your circuit breaker backfires — outlier detection in Istio and Envoy

CloudCon Sydney 2026 presentation
https://cloudcon.au/#talk=when-your-circuit-breaker-backfires-outlier

1. When your circuit breaker backfires — outlier detection in Istio and Envoy Mitsuhiro Shibuya / @m4buya Mercari, Inc.
2. Istio has a feature called outlier detection. Istio’s docs have a guide called Circuit Breaking. It covers two different things: • • connectionPool: limits on connections and requests. The caller protects itself outlierDetection: watch the responses, and take instances that look bad out of the pool ◦ Envoy calls it passive health checking: guessing health from responses The second one is today’s subject. This is how it works. 2
3. Outlier detection: how it works. load-balancing pool 3 cap: P% of the pool requests caller's Envoy outlier detection 200 … 503 3 1 pod pod pod ejected 0 in a row 4 in a row out for T × 1 2 consecutive5xxErrors 1 eject a pod after N 5xx responses in a row 2 keep it out for T × (number of times it has been ejected) baseEjectionTime 3 no more than P% out at once — past that, it can't eject maxEjectionPercent
4. The team had a problem. Under heavy load, workers got stuck. Pods stopped answering and were restarted. “If we take a stuck pod out early, things should get better.” Istio’s docs have a page called Circuit Breaking. It looked like the right tool. 4
5. The setting went live. The incident started. Not a full outage. The error rate went up and stayed up. • • The services calling it were affected too, not only the service with the setting Six hours until recovery The cause was the setting that was meant to protect the service. 5
6. Nothing about it looked wrong. outlierDetection: consecutive5xxErrors: 20 interval: 30s baseEjectionTime: 30s maxEjectionPercent: 5 • • • • Based on the official docs, safe-looking values The PR said: “primarily to observe the effects” Normal change process. Roll out slowly, watch, roll back if needed Even now, I see no reason to block this change 👉 This is not a story about someone’s fault. 6
7. What production looked like. every pod of the service, one line each · production data, pod names removed 7
8. The same pods, averaged by zone. the three zones have the same number of pods · production data 8
9. Why one zone? The breaker changed the routing. “Configuring OutlierDetection in a DestinationRule will automatically activate locality-aware failover.” • • • Failover needs a health signal. The breaker is that signal. So writing the breaker turns the local routing on Every caller now sends to its own zone first. The biggest caller had most of its pods in one zone A busy pod is still healthy. So failover never moved the load The opt-out is one line: localityLbSetting.enabled: false. Nobody knew it had to be written. 9 Zones caller failover target target
10. Four layers. Each one doing its job. callers give up and retry: the load becomes 2–3.5× Callers Service mesh outlierDetection goes in, and zone-local routing comes with it Application ⅓ of the pods take most of the load worker pool fills, requests slow down Kubernetes liveness probe times out, then restart 10 locality failover never happens; outlierDetection ejects a few pods fewer pods in the zone receive the same load
11. Everything worked as designed. That’s why it broke. 11
12. Could the breaker have softened this? I tried. We thought of outlierDetection as a circuit breaker. Could it have softened this? A small experiment: twenty pods, one caller, four pods broken on purpose. • • • With the incident settings, the breaker barely acted. The cap allowed one ejection With aggressive settings, it acted. And things got worse: 5-35x more pods restarted, not fewer The reason: it ejects busy pods along with broken ones. A 503 from a full request queue looks the same as a 503 from a broken pod I could not find a setting that helped on its own. OutlierDetection 🧐 helped only together with fixes outside the mesh. 12
13. You need to break the loop. Possible fixes Limit the number of retries callers give up and retry: the load becomes 2–3.5× Callers Right-size maxEjectionPercent Service mesh outlierDetection goes in, and zone-local routing comes with it locality failover never happens; outlierDetection ejects a few pods Disable zone-local routing Application Kubernetes ⅓ of the pods take most of the load Return 429 when just busy worker pool fills, requests slow down liveness probe times out, then restart Let liveness probe pass when busy 13 fewer pods in the zone receive the same load Reserve per-zone capacity
14. In production, the team did these two. Limit the number of retries callers give up and retry: the load becomes 2–3.5× Callers Right-size maxEjectionPercent Service mesh outlierDetection goes in, and zone-local routing comes with it locality failover never happens; outlierDetection ejects a few pods Disable zone-local routing Application Kubernetes ⅓ of the pods take most of the load Return 429 when just busy worker pool fills, requests slow down liveness probe times out, then restart Let liveness probe pass when busy 14 fewer pods in the zone receive the same load Reserve per-zone capacity
15. Resilience has to be designed as a whole. One setting, in one layer. The fix was never in that one layer. That is not an accident. Every control makes assumptions about the layers around it: how retries help, what an error means, how much capacity is left. So the work is not tuning one control. It is seeing how the controls fit together, because that is where the loop is, and where it is cut. 15
16. Observe first. In this incident, nothing showed that a control had acted. • • • Turn on the stats for the controls you already have, before you change one. The outlier detection's counters are off by default: ask for them Run the new control in observe mode where there is one. Envoy's outlier detection has it; Istio's DestinationRule does not expose it Where there is none, observe it in a small experiment first. Twenty pods and one load generator found both surprises in this talk Then enforce, with the signals already on the dashboard. 16
17. Takeaways Everything worked as designed. That’s why it broke. • • • Istio's outlierDetection activates zone-local routing Resilience has to be designed as a whole. The setting was in one layer; the fix was never in that layer Observe first: stats on, observe mode or a small experiment, then enforce The problem was what nobody designed: the whole. 17
18. Thank you! Connect on We're hiring!