Failing Gracefully as a Feature

The move from monolith to microservice has allowed pieces of functionality to be deployed individually and on demand. Having functionality isolated allows the opportunity for one microservice to fail without bringing down the whole system.

However, it also increases complexity with the number of API calls being made across all of these services. Each service has unique failure models, whether its a database, cache, queue, etc. How can you be sure that one single failure doesn’t cause an outage for your end users?

Landing the launches of new products and features and providing your users with a positive experience is crucial to your success. If something is to fail, you’d prefer they didn’t know. Or if they did, it shouldn’t interrupt their experience.

In this talk, we’ll cover graceful degradation as an engineering goal which can be confidently tested with Chaos Engineering. By purposely causing failure of one service at a time in a controlled environment, you can safely observe the effect on the end user, whether that’s on a laptop browser, a mobile app, or the result of an API call.

Transcript

1. Failing Gracefully As A Feature Lorne Kligerman Director of Product, Gremlin @lklig
4. 4 Be down in 10! T-Ho 2017 Hey team… bit of a spill but I’m fine.
5. 5 We Expect Technology To Just Work™
6. Technical Issues Likely Cost Retailers Billions 12.01.16 Macy’s, Lowe’s hit by Black Friday technical glitches 11.27.17 Retail outages online leave shoppers frustrated on Black Friday 11.23.18 People.com Black Friday Failures @lklig
7. Wells Fargo accidentally foreclosed hundreds of homeowners 8.7.18 Customers report difficulty accessing Chase Bank mobile and online 2.16.19 Citibank Website down, not working 2.28.19 Investopedia Breaking Banks @lklig
8. Computer Problems Blamed For Flight Delays 4.1.19 Major US Airlines hit by delays after glitch at vendor 4.1.19 Pilots of doomed Boeing 737 MAX fought the plane’s software and lost 4.4.19 Airline Incidents @lklig
9. 9 Technology is fragile. When it breaks, we shouldn’t notice. @lklig
10. 10 Plan ahead to keep your users happy FAILURE GRACEFUL DEGRADATION @lklig
11. 11 Why Are Failures So Common?
12. 12 Legacy Systems @lklig
13. 13 Lack of Testing Failure UI End to end Integration Unit @lklig
14. 14 With Scale Comes Complexity @lklig
15. @lklig
16. @lklig
17. 17 What Can We Do About It?
18. 18 Design For Failure
19. 19 @lklig
20. 20 Designing For Failure Key User Stories & Features Edge Cases From Unexpected User Behaviour Dependency Failures @lklig
21. 21 21 @lklig
22. 22 @lklig Loading Screens Are Not Graceful
23. 23 Inject Failure By Breaking Things On Purpose @lklig
24. Inject failure one service at a time. Maintain critical functionality. 24 @lklig
25. Common Failures Modes 25 Errors HTTP 400, 401, 402 500, 503 Blackhole Latency @lklig THAT DEGRADE THE USER EXPERIENCE @lklig
26. 26 Degrade Gracefully
27. 27 Graceful Degradation • Provide the best possible experience • All but the most critical functionality can fall off • Don’t give up on your users, hold state as long as possible @lklig
28. 28 @lklig When one dependency fails, users are often affected Storage Auth User Data Content Cache Feature 1 Feature 2
29. 29 Implemented As Designed @lklig
30. 30 Added Latency @lklig
31. 31 Blocked Video Link @lklig
32. 32 Blocked JQuery Request @lklig
33. 33 @lklig
34. 34 Delight Your Users
35. 35 35 Graceful Degradation Done Right @lklig
36. 36 Positive Business Impact Product Launch Delight users with new features Success Metrics Quantitative goals of the launch Product Landing Were the goals achieved? Why or why not? What got in way? @lklig
37. 37 @lklig Maintain release velocity Deliver a positive user experience Engineers spend less time in war rooms Plan Experiments Early @lklig
38. 38 RELIABILITY THROUGH CHAOS ENGINEERING Design for Failure Identify the most critical end user functionality. Inject Failure Impact your system to be sure your user experience isn’t impacted. Degrade Gracefully Plan for non critical functionality not to get in the way. Delight Your Users Your product metrics will show behaviour, no matter the condition. Graceful Degradation As a Feature @lklig
39. USE LORNE FOR 20% OFF
40. 40 gremlin.com/lorne
41. Q&A Lorne Kligerman Director of Product, Gremlin @lklig