Tweet
Tweet

Slide 1

Slide 1 text

EMBRACING FAILURES TO PREVENT MELTDOWN RUBY X ELIXIR CONF TW, 2018

Slide 2

Slide 2 text

I AM THE FOUNDER AND CEO OF Ruby on Rails dev shop based in Kochi, India We love working in Ruby on Rails and the ruby community.

Slide 3

Slide 3 text

{ “name” => "Harisankar P S", “email” => ”[email protected]”, “twitter” => "coderhs", “facebook" => "coderhs", “github” => “coderhs”, “linkedin” => “coderhs”, } I have been the organiser of @keralarb Meetups since 2012,
 Rails Girls Kochi, and Open Source Saturday kochi. I am a Ruby, Elixir and Crystal Developer I volunteer and has been local organiser of Ruby Conf India, 2016 & 2017

Slide 4

Slide 4 text

I am from Kochi, Kerala, India.

Slide 5

Slide 5 text

Interesting fact about India: India has 22 official languages, 1653 spoken language and over 50,000 Dialects And I know 3 of them

Slide 6

Slide 6 text

I am love being in # its my second visit here - the places is good, the people are good, the ruby community is great,
 the is amazing, and food is delicious.

Slide 7

Slide 7 text

I want to start with a question?

Slide 8

Slide 8 text

HARDWARE OR SOFTWARE?

Slide 9

Slide 9 text

Hardware or Software which do you think is better?

Slide 10

Slide 10 text

Imagine a firewall, is software or hardware better?

Slide 11

Slide 11 text

Me, being a fan of programming I instantly said
 software, because as its easy to control and extend
 using algorithms

Slide 12

Slide 12 text

Yes when someone hacks the system, we can use software,
 and try blocking him using algorithms, filters, etc Or

Slide 13

Slide 13 text

You can cut the lan cable using a scissors and prevent anyone from hacking

Slide 14

Slide 14 text

The reason why I told this story is to explain that
 many things that we build into our software
 is based on real world entities.
 
 Thats whats object oriented programming is about So I am going to talk about two such patterns today. Timeout patterns and Circuit breaker pattern.

Slide 15

Slide 15 text

Keeping that in mind. Lets Start!

Slide 16

Slide 16 text

CIRCUIT BREAKER PATTERN

Slide 17

Slide 17 text

APP A APP B APP C Dependency Chain

Slide 18

Slide 18 text

▸ Wrap a method call inside a circuit breaker object. ▸ This object monitors the success rate of your piece of code ▸ If it fails often, one after the other - it breaks the circuit ▸ The mantra here is to fail FAST!

Slide 19

Slide 19 text

No content

Slide 20

Slide 20 text

▸ Its basically about being bold when it comes to handling failures ▸ Our app will fail, it will break down or the API service we use might break down. ▸ There is no 100%

Slide 21

Slide 21 text

EXAMPLE: A GOOGLE RANK TOOL (NON MICROSERVICE EXAMPLE)

Slide 22

Slide 22 text

APPLICATION 3RD PARTY RANK CHECKING TOOL Every day you send 100,000
 Keywords and ranks to check 
 their rank, one by one You get back a response when
 they are ready.

Slide 23

Slide 23 text

APPLICATION 3RD PARTY RANK CHECKING TOOL But today the service has a faulty
 hard drive and the application is slow
 and you still send it more request. You get back nothing.

Slide 24

Slide 24 text

YOU GET CALLED TO COME ONLINE AT 3 AM IN THE MORNING

Slide 25

Slide 25 text

IN MICRO SERVICES : IT PREVENTS SNOWBALL EFFECT ▸ One service slowed down by 200 ms ▸ Second service slowed down by another 400 ms ▸ when it reaches the file place it slowed down by 1000-5000ms

Slide 26

Slide 26 text

WHAT CONSTITUTES A FAILURE ▸ Not performing the required action ▸ Giving a different result other than the one expected ▸ Taking more than the acceptable time to get a response. Here we break to talk about timeout

Slide 27

Slide 27 text

TIMEOUT ▸ We need to have checks and balances throughout the software ▸ We usually think about it when we break our app into microservices, where we give specification to another developer. ▸ But we need to think our software as well: ▸ In a DB Query ▸ HTTP Request (Ruby waits for 2 seconds by default)

Slide 28

Slide 28 text

TEXT ▸ Timeouts improve fault isolation ▸ Timeouts forces us to use our resources properly ▸ Timeouts are mandatory if your mantra is Fail Fast

Slide 29

Slide 29 text

3 STATES OF A BREAKER

Slide 30

Slide 30 text

No content

Slide 31

Slide 31 text

CLOSED STATE ▸ This is when the system lets everything pass. The encapsulated code is executed ▸ There are no failures happening or its less not enough to cause you a problem ▸ We are all happy Just summarising again

Slide 32

Slide 32 text

OPEN STATE ▸ When we cross a limit of the number of failures, we open the circuit ▸ Which means no more requests will reach the service or underlying code ▸ The system will provide it with an alternate route ▸ cache entry ▸ add it to the retry queue ▸ Just show an appropriate error.

Slide 33

Slide 33 text

HALF OPEN ▸ This state is when the system is trying to see the underlying code is fixed ▸ It sends one of the requests through, after a threshold time and if it succeeds. Then try few more requests or an existing failed request, and finally closes the circuit and send all requests through ▸ So right now the system is sort of self-healing

Slide 34

Slide 34 text

TEXT QUANTIFIABLE ANALYTICS ▸ 100 ms - 600 ms, # of query - your non tech team members have no idea what it means ▸ Measure in number of failures per minute/per hour/per day ▸ We can analyze code improvement by measuring improvement # of code failures, in production. ▸ We can deploy to one server out of many, monitors the performance in that server and then rollback or mass deploy automatically

Slide 35

Slide 35 text

TEXT IMPROVE PERFORMANCE: TIMEOUT + CIRCUIT BREAKERS 80 MS 100 MS 150 MS 75 MS Micro Services with timeouts that breaks circuits. 80 MS 100 MS 125 MS 75 MS Reduce the timeout time and see how much the failure increases and start to fine tuning

Slide 36

Slide 36 text

▸ Circuit breakers give you a configurable policy ▸ No Resource Build UP - 10K req/seconds with 5 seconds timeout thats 50K requests in the pipeline ▸ We can build Monitoring systems using this - see whats breaking

Slide 37

Slide 37 text

GEMS FOR IT IN RUBY?

Slide 38

Slide 38 text

Build your own

Slide 39

Slide 39 text

No content

Slide 40

Slide 40 text

https://github.com/wsargent/circuit_breaker

Slide 41

Slide 41 text

https://github.com/pedro/cb2

Slide 42

Slide 42 text

https://github.com/jnunemaker/resilient

Slide 43

Slide 43 text

LIBRARIES FOR IT IN ELIXIR?

Slide 44

Slide 44 text

https://github.com/awochna/breaker

Slide 45

Slide 45 text

https://github.com/mmzeeman/breaky

Slide 46

Slide 46 text

https://github.com/klarna/circuit_breaker

Slide 47

Slide 47 text

TO SUMMARIZE

Slide 48

Slide 48 text

SEEING THIS REALLY FAST

Slide 49

Slide 49 text

IS BETTER THAN

Slide 50

Slide 50 text

SEEING THIS AFTER WAITING FOR 1 MINUTE

Slide 51

Slide 51 text

Thank You | 谢谢