Bulletproof API's

In high-availability systems is necessary to deal with inevitable failures. Whether they are temporary or not, we can have a direct impact in the user experience.

Bulletproof API's is a set of practices, techniques, and implementations that focus in providing resilience, identifying the type of failure and making possible to recover quickly, reducing or even eliminating the impact in the user experience.

Some topics that will be covered:
* Retries
* Circuit Breakers
* Timeouts
* How to identify and deal with different types of traffic
* Effective metrics analysis
* A real case and how the techniques were applied

At the end of the talk, it will be possible to understand how resilience approaches can impact the improvement of the users final experience.
An objective view of tools that we have available in the market.

And the importance of identifying temporary failures and how to deal with them.

Transcript

1. 1 Bulletproof API's
2. 2 Vinicius Souza. Software Engineer at @globocom github.com/vsouza @iamvsouza
3. 3 What is a bulletproof thing? " d e s c r i b e a n o b j e c t a s b e i n g indestructible, to never stop working, or to outperform other similar products."
4. 4 Where? failure points
5. 5 East West traffic. East-west traffic refers to the network communications between servers, applications and databases within the data center. Service Two Service One Database
6. 6 North South. North-South traffic is the client server traffic that goes between users in a branch office and the data center that hosts the application that they are accessing. Web App User
7. 7 Heavy traffic. Only 20% a datacenter traffic is a North South. 20% 80% East West North South
8. 8 When? failure moments
9. 9 Now!
10. 10 Timeouts. Stop waiting for an answer.
11. 11 Humans should not be involved in setting timeouts. Ben Christensen - Facebook
12. 12 Percentiles. Sample of ascending order values, divided into one hundred parts.
13. Calculating percentiles. 953 ms | 99% 497ms | 70% 13 E X : 1 4 9 , 1 8 5 , 2 2 1 , 2 7 3 , 3 0 9 , 4 3 0 , 4 5 1 , 4 9 7 , 8 2 1 , 9 5 3 430ms | 50%
14. 14 More than "now"!
15. 15 Retries mechanisms can have success after a simple failure and most important, can help us to identify which kind of failure we have. Retry. Backoffs Max Attempts Identify
16. 16 Retry example Elixir - Tesla HTTP Client github.com/teamon/tesla
17. 17 Long "now"!
18. 18 Circuit breakers. Wrap a protected function call in a circuit breaker object, which monitors for failures. Once the failures reach a certain threshold, the circuit breaker trips, and all further calls to the circuit breaker return with an error or with some alternative service or default message, without the protected call being made at all.
19. 19 Start monitoring for errors. If threshold was reached the circuit change the state. Start monitoring for recovering. If have a confirmed success, alter circuit breaker state. Function wrapped
20. 20 Circuit closed. All dependencies online and ok. E X T 1 C A C H E
21. 21 Circuit open. Stop calling dead dependency. E X T 1 C A C H E X
22. 22 Fallback. Return an error or a default message. E X T 1 C A C H E X Fallback X
23. 23 Circuit Breaker example Erlang Fuse library github.com/jlouis/fuse
24. 24 Real World. FIFA WORLD CUP RUSSIA 2018
25. 25 Streaming audience. 2006 2010 2014 550.000 240.000 44.000 World Cup simultaneous users
26. 26 Authorize API. True || False
27. 27 BRA X COS. 22/06/2018 - 09:00:00 5' 6' 7' 8' 9' 10' 0 1 2.092 7.580 2 0 Rescued errors by Circuit Breaker
28. 28 POR X ESP. 15/06/2018 - 03:00:00 PM 0' 8' 24' 44' 55' 58 88 70 673 444 444 217 228 2 68.300 50.100 54.700 34.300 34.100 33.100 37.600 Req/m Rescued errors by Circuit Breaker
29. 29 KOR X GER. 22/06/2018 - 11:00:00 63' 64' 65' 66' 34.900 86.000 72.100 52.800 Req/m 63' 64' 65' 66' 125 141 107 101 service 1 P95
30. 30 1.200.000 simultaneous users *New record
31. 31 We are hiring! talentos.globo.com
32. 32 Opensource! opensource.globo.com
33. 33 References. github.com/Netflix/Hystrix/wiki github.com/teamon/Tesla bit.ly/percentis https://dzone.com/articles/circuit-breaker-fallback-and-load-balancing-with-n https://speakerdeck.com/jboner/without-resilience-nothing-else-matters https://dzone.com/articles/understanding-retry-pattern-with-exponential-back https://www.networkworld.com/article/2177684/the-growth-in-east-west-traffic.html https://blog.advaoptical.com/en/north-south-east-west-all-in-the-cloud
34. 34 THANK YOU.