Fault-Tolerant Clients: Escrevendo Clients e Services Tolerantes a Falhas

@RestController
public class CalculadoraDeFretesController
{

@GetMapping(path="/fretes/calcula")
public ResponseEntity calcula(@RequestParam String cep)
{

// consulta frete em outro microsserviço
String url = "https://ms.fast-fretes.com/calcula-frete/"
;

 
ZupHttpClient client = new ZupHttpClient<>()
;

Frete frete = client.get(url
)

.withParameter("cep", cep
)

.execute(); 
return ResponseEntity.ok(frete);

}

}

View Slide

@RestController
{

{

;

 
;

)

)

.execute(); 

}

}

View Slide

@RestController
{

{

;

 
;

)

)

.execute(); 

}

}

View Slide

@RestController
{

{

;

 
;

)

)

.execute(); 

}

}

View Slide

@RestController
{

{

;

 
;

)

)

.execute(); 

}

}

View Slide

@RestController
{

{

;

 
;

)

)

.execute(); 

}

}

View Slide

@RestController
{

{

;

 
;

)

)

.execute(); 

}

}

View Slide

@RestController
{

{

;

 
;

)

)

.execute(); 

}

}

View Slide

@RestController
{

{

;

 
;

)

)

.execute(); 

}

}

View Slide

@RestController
{

{

;

 
;

)

)

.execute(); 

}

}

View Slide

@RestController
{

{

;

 
;

)

)

.execute(); 

}

}

View Slide

@RestController
{

{

;

 
;

)

)

.execute(); 

}

}

View Slide

Design for
Failures

View Slide

Rafael Ponte

@rponte

View Slide

Fortaleza - Terra do Sol

View Slide

FAULT-TOLERANT CLIENTS
Implementando clients e services mais resilientes e
tolerantes a falhas

View Slide

Implementação
Ingênua

View Slide

@RestController
{

{

;

 
;

)

)

.execute(); 

}

}

View Slide

@RestController
{

{

;

 
;

)

)

.execute(); 

}

}

View Slide

@RestController
{

{

;

 
;

)

)

.execute(); 

}

}

;

)

)

.execute();

;

)

)

.execute();

;

)

)

.execute();

;

)

)

.execute();
Pode ser
qualquer
HTTP Client!

;

)

)

.execute();

;

)

)

.execute();

;

)

)

.execute();

O problema dessa
implementação…

View Slide

Spring Boot App
Client
Spring Boot App
Service
 
Internet

View Slide

Spring Boot App
Client
Spring Boot App
Service
 
Internet

View Slide

Spring Boot App
Client
Spring Boot App
Service
 
Internet

View Slide

Spring Boot App
Client
Spring Boot App
Service
 
Internet

View Slide

Spring Boot App
Client
Spring Boot App
Service
 
Internet

View Slide

Spring Boot App
Client
Spring Boot App
Service
 
Internet

View Slide

Spring Boot App
Client
Spring Boot App
Service
 
Internet

View Slide

Spring Boot App
Client
Spring Boot App
Service
 
Internet

View Slide

Spring Boot App
Client
Spring Boot App
Service
 
Internet

View Slide

Spring Boot App
Client
Spring Boot App
Service
 
Internet

View Slide

Spring Boot App
Client
Spring Boot App
Service
 
Internet

View Slide

Spring Boot App
Client
Spring Boot App
Service
 
Internet

View Slide

Spring Boot App
Client
Spring Boot App
Service
 
Internet

View Slide

Spring Boot App
Client
Spring Boot App
Service
 
Internet

View Slide

Spring Boot App
Client
Spring Boot App
Service
 
Internet

View Slide

Spring Boot App
Client
Spring Boot App
Service
 
Internet

View Slide

Spring Boot App
Client
Spring Boot App
Service
 
Internet

View Slide

Tudo isso porque
esperamos demais…

View Slide

Timeout

View Slide

ZupClientConfig config = // ... ZupClientConfig.custom(
)

.withRequestTimeout(Duration.of(5, SECONDS)
)

.build()
;

;

)

)

.execute();

View Slide

ZupClientConfig config = // ...
)

.build()
;

;

)

)

.execute();

View Slide

ZupClientConfig config = // ...
)

.build()
;

ZupHttpClient client = new ZupHttpClient<>(config)
;

)

)

.execute();

View Slide

ZupClientConfig config = ZupClientConfig.custom(
)

)

.build()
;

;

)

)

.execute();

View Slide

)

)

.build()
;

;

)

)

.execute();

View Slide

)

)

.build()
;

;

)

)

.execute();

View Slide

)

)

.build()
;

;

)

)

.execute();
Depende do caso de
uso e contexto

View Slide

)

)

.build()
;

;

)

)

.execute();

View Slide

Spring Boot App
Client
Spring Boot App
Service
 
Internet

View Slide

Spring Boot App
Client
Spring Boot App
Service
 
Internet

View Slide

Spring Boot App
Client
Spring Boot App
Service
 
Internet

View Slide

Spring Boot App
Client
Spring Boot App
Service
 
Internet

View Slide

Spring Boot App
Client
Spring Boot App
Service
 
Internet

View Slide

Fail Fast

View Slide

…mas não para
Transient Failures

View Slide

Spring Boot App
Client
Spring Boot App
Service
 
Internet

View Slide

Spring Boot App
Client
Spring Boot App
Service
 
Internet

View Slide

Spring Boot App
Client
Spring Boot App
Service
 
Internet

View Slide

Spring Boot App
Client
Spring Boot App
Service
 
Internet

View Slide

Spring Boot App
Client
Spring Boot App
Service
 
Internet

View Slide

Spring Boot App
Client
Spring Boot App
Service
 
Internet

View Slide

Não podemos jogar
esse problema no
usuário…

View Slide

Retry

View Slide

)

)

.build()
;

;

)

)

.execute();

View Slide

)

)

.withRetryPolicy(
)

.build()
;

;

)

)

.execute();

View Slide

)

)

.withRetryPolicy(RetryPolicy.custom().build()
)

.build()
;

;

)

)

.execute();

View Slide

)

)

.withRetryPolicy(RetryPolicy.custom(
)

.retryOn(HttpStatus5xxException.class
)

.build()
)

.build()
;

;

)

)

.execute();

View Slide

)

)

)

)

.build()
)

.build()
;

;

)

)

.execute();
Transient Failures

View Slide

)

)

)

)

.withMaxAttempts(3
)

.build()
)

.build();
;

)

)

.execute();

View Slide

)

)

)

)

.withMaxAttempts(3
)

.build()
)

.build();
;

)

)

.execute();
Depende do caso de
uso e contexto

View Slide

Spring Boot App
Client
Spring Boot App
Service
 
Internet

View Slide

Spring Boot App
Client
Spring Boot App
Service
 
Internet

View Slide

Spring Boot App
Client
Spring Boot App
Service
 
Internet
Original Request:

View Slide

Spring Boot App
Client
Spring Boot App
Service
 
Internet
Original Request:

View Slide

Spring Boot App
Client
Spring Boot App
Service
 
Internet
Retry #1:
Original Request:

View Slide

Spring Boot App
Client
Spring Boot App
Service
 
Internet
Retry #1:
Retry #2:
Original Request:

View Slide

Spring Boot App
Client
Spring Boot App
Service
 
Internet
Retry #1:
Retry #2:
Original Request:

View Slide

Clients são
EGOISTAS

View Slide

Spring Boot App
Client
Spring Boot App
Service
 
Internet

View Slide

Spring Boot App
Client
Spring Boot App
Service
 
Internet

View Slide

Spring Boot App
Client
Spring Boot App
Service
 
Internet

View Slide

Spring Boot App
Client
Spring Boot App
Service
 
Internet

View Slide

Spring Boot App
Client
Spring Boot App
Service
 
Internet

View Slide

Spring Boot App
Client
Spring Boot App
Service
 
Internet

View Slide

Spring Boot App
Client
Spring Boot App
Service
 
Internet

View Slide

Spring Boot App
Client
Spring Boot App
Service
 
Internet

View Slide

RETRY STORM
 
Vamo surrar esse servidor ae!

View Slide

Não demos
tempo pro serviço
se recuperar

View Slide

Retry with
Backo
f

View Slide

)

)

)

)

.withMaxAttempts(3
)

.build()
)

.build()
;

;

)

)

.execute();

View Slide

)

)

)

)

.withMaxAttempts(3
)

.withDelay(Duration.of(100, MILLIS)
)

.build()
)

.build();
;

)

)

.execute();

View Slide

100ms 
100ms 
100ms 
100ms 
… 
Retry #1
Retry #2
Retry #3
Retry #4

View Slide

Não resolvemos o
problema, somente
"empurramos" ele
100ms pra frente…

View Slide

)

)

)

)

.withMaxAttempts(3
)

.withDelay(Duration.of(100, MILLIS)
)

.build()
)

.build();
;

)

)

.execute();

View Slide

)

)

)

)

.withMaxAttempts(3
)

.withDelay(Duration.of(100, MILLIS), 2
)

.build()
)

.build();
;

)

)

.execute();

View Slide

)

)

)

)

.withMaxAttempts(3
)

)

.build()
)

.build()
;

;

)

)

.execute();
Delay Factor

View Slide

100ms 
200ms 
400ms 
800ms 
… 
Retry #1
Retry #2
Retry #3
Retry #4
Dobramos
Dobramos
Dobramos
Dobramos

View Slide

Retry with
Exponential
Backo
f

View Slide

)

)

)

)

.withMaxAttempts(3
)

)

.build()
)

.build()
;

;

)

)

.execute();

View Slide

)

)

)

)

.withMaxAttempts(3
)

)

.build()
)

.build()
;

;

)

)

.execute();

View Slide

)

)

)

)

.withMaxAttempts(3
)

)

.build()
)

.build()
;

;

)

)

.execute();

View Slide

)

)

)

)

.withMaxAttempts(3
)

.withExponentialBackoff(Duration.of(100, MILLIS)
)

.build()
)

.build()
;

;

)

)

.execute();

View Slide

Demos folêgo pro
servidor ter tempo para
se recuperar, porém…

View Slide

SINCRONIA
ENTRE CLIENTS

View Slide

Precisamos diminuir
as chances dessa
sincronização…

View Slide

Retry with
Backo
f
f
and
Jitter

View Slide

jitter /ˈdʒɪtə/ n.
 
the deviation from true
periodicity of a presumably
periodic signal.

View Slide

jitter /ˈdʒɪtə/ n.
 
add randomness to the
periodicity of a periodic
signal.

View Slide

)

)

)

)

.withMaxAttempts(3
)

)

.build()
)

.build(); 
;

)

)

.execute();

View Slide

)

)

)

)

.withMaxAttempts(3
)

)

.withJitter(0.25
)

.build()
)

.build()
;

;

)

)

.execute();

View Slide

)

)

)

)

.withMaxAttempts(3
)

)

.withJitter(0.25
)

.build()
)

.build()
;

;

)

)

.execute();
Jitter Factor

View Slide

delay random(0, delay*0.25)
+
-

+
-

+
-

+
-

Jitter
+
-

100ms 
200ms 
400ms 
800ms 
… 
Retry #1
Retry #2
Retry #3
Retry #4

View Slide

121ms
+21ms
 
203ms
+3ms
 
393ms
-7ms
 
788ms
-12ms
 
… 
Retry #1
Retry #2
Retry #3
Retry #4

View Slide

Não só evitamos a
sincronização…

View Slide

… como também
distribuímos
melhor os requests
entre os intervalos

View Slide

Retry with Exponential Backo
f
https://aws.amazon.com/blogs/architecture/exponential-backo
f
f
-and-jitter/

View Slide

f
f
f
-and-jitter/
Intervalos longos
 
e ociosos

View Slide

Intervalos longos
 
e ociosos
f
Picos ainda
podem ocorrer
f
f
-and-jitter/

View Slide

f
f
and Jitter
f
f
-and-jitter/

View Slide

f
f
and Jitter
f
f
-and-jitter/
Distribui as requisições entre os intervalos

View Slide

f
f
and Jitter
Menor
incidência
de picos
f
f
-and-jitter/

View Slide

f
f
and Jitter
Menor
incidência
de picos
f
f
-and-jitter/

View Slide

Pra entender
melhor como tudo
isso funciona…

View Slide

Efeitos de
Particionamento
de Rede

View Slide

RetryPolicy.custom(
)

)

.withMaxAttempts(3
)

.withDelay(ofMillis(100))
.build(
)

(Rede particionada)
https://www.usenix.org/sites/default/
f
i
les/conference/protected-
f
i
les/srecon18asia_slides_goh.pdf

View Slide

RetryPolicy.custom(
)

)

.withMaxAttempts(3
)

.withExponentialBackoff(ofMillis(100))
.build(
)

f
i
f
i
(Rede particionada)

View Slide

RetryPolicy.custom(
)

)

.withMaxAttempts(3
)

.withJitter(0.25) 
.build()
f
i
f
i
(Rede particionada)

View Slide

Efeitos de
falhas parciais

View Slide

RetryPolicy.custom(
)

)

.withMaxAttempts(3
)

.withDelay(ofMillis(100))
.build(
)

(taxa de falha de 25 %)
f
i
f
i

View Slide

RetryPolicy.custom(
)

)

.withMaxAttempts(3
)

.build(
)

f
i
f
i

View Slide

RetryPolicy.custom(
)

)

.withMaxAttempts(3
)

.withJitter(0.25) 
.build()
f
i
f
i

View Slide

Concluindo…

View Slide

Sistemas
Distribuídos são
complexos por
natureza

View Slide

Qualquer chamada
remota é um
potencial ponto de
falha

View Slide

Pontos de atenção

View Slide

• Sempre de
f
i
na Timeouts

Pontos de atenção

View Slide

• Sempre de
f
i
na Timeouts

• Não faça Retry (por default)

Pontos de atenção

View Slide

• Sempre de
f
i
na Timeouts


• Se
f
i
zer Retry, faça Backo
f
f
(exponential)

Pontos de atenção

View Slide

• Sempre de
f
i
na Timeouts


• Se
f
i
zer Retry, faça Backo
f
f
(exponential)

• Sempre use Jitter
Pontos de atenção

Fault-Tolerant Clients: Escrevendo Clients e Services Tolerantes a Falhas

Fault-Tolerant Clients: Escrevendo Clients e Services Tolerantes a Falhas

Rafael Ponte

More Decks by Rafael Ponte

Other Decks in Technology

Featured

Transcript