Resiliency and Availability Design Patterns for the Cloud

© 2021, Amazon Web Services, Inc. or its Affiliates.
Resiliency and Availability
Design Patterns for the Cloud
Sébastien Stormacq
Principal Developer Advocate
@sebsto

View Slide

© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Can you guess what will happen?

View Slide

Distributed Systems are hard

View Slide

Failures are a given and
everything will eventually
fail over time.
Werner Vogels
CTO – Amazon.com
“
“

View Slide

Resiliency: Ability for a system to handle and
eventually recover from unexpected conditions

View Slide

Partial failure mode

View Slide

People
Application
Network & Data
Infrastructure

View Slide

Let’s talk about Geo Availability

View Slide


View Slide

Fully-scaled Availability Zone

View Slide

Highly redundant regional network

View Slide

AWS Region and availability zones
Region
Availability zone a Availability zone b Availability zone c
data center
data center
data center
1 or more data centers per AZ
2 or more AZs per region (new regions min 3)
data center
data center
data center
data center
data center
data center

View Slide

Availability in parallel
Component Availability Downtime
X 99% (2-nines) 3 days 15 hours
Two X in parallel 99.99% (4-nines) 52 minutes
Three X in parallel 99.9999% (6-nines) 31 seconds

View Slide

Multi-AZ architecture
Region
Instances Instances Instances
DB Instance DB instance
standby
Elastic Load
Balancing
(ELB)

View Slide

X
Region
standby
Elastic Load
Balancing
(ELB)

View Slide

Region
standby
Elastic Load
Balancing
(ELB)
X

View Slide

Region
new master
Elastic Load
Balancing
(ELB)
X

View Slide

• Enables fault-tolerant applications
• AWS regional services designed to
withstand AZ failures
• Leveraged by AWS regional
services such as Amazon S3,
Amazon DynamoDB, Amazon
Aurora, Amazon ELBs, etc.
Region
standby
Elastic Load
Balancing (ELB)

View Slide

Let’s talk about auto scaling

View Slide

Auto-Scaling
Fixed
Variable

View Slide

Availability zone 1
Auto Scaling group
AWS Region
Availability zone 2
Auto-scaling for self-healing
Elastic Load
Balancing
(ELB)
X

View Slide

Let’s talk about decoupling and async

View Slide

Process A Process B Process A Process B
Synchronous Asynchronous
Waiting
Working
Continues
get or fetch result
Get result
Decoupling with async pattern

View Slide

API: {DO foo}
PUT JOB: {JobID: 0001, Task: DO foo}
API: {JobID: 0001}
GET JOB: {JobID: 0001, Task: DO foo}
{JobID: 0001, Result: bar}
Cache node
Worker
Instance
Worker
Instance
Queue/Streaming
API
Instance
API
Instance
API
Instance

View Slide

Push Notification
User
Worker
Instance
Worker
Instance
API
Instance
API
Instance
Cache node
Fetch results
API
Instance
Queue/Streaming

View Slide

Degrade & prioritize traffic
with queues
Worker
Instance
Worker
Instance
API
Instance
API
Instance
API
Instance
High Priority Queue
Low Priority Queue

View Slide

Let’s talk about databases.

View Slide

Read / Write separation
read replica
DB instance
read replica
DB instance
read replica
Instance Instance
Instance
Supports degradation through Read-Only mode

View Slide

Database Federation
Users
DB
Products
DB
Instance Instance
Instance
DB Instance
DB instance
read replica
DB Instance
DB instance
read replica

View Slide

Database Sharding
User ShardID
002345 A
002346 B
002347 C
002348 B
002349 A
C
B
A
Instance Instance
Instance
DB Instance
DB instance
read replica
DB Instance
DB instance
read replica
DB Instance
DB instance
read replica

View Slide

Let’s talk about timeouts, backoff &
retries!

View Slide

Users
App
DB
Conn
Pool
INSERT
INSERT
INSERT
INSERT
What happens if the DB “slows down”?
Timeout client side Timeout backend side
?
?

View Slide

User 1
App DB
Conn
Pool
INSERT
Timeout client side = 10s Timeout backend side = default = Infinite
Retry INSERT
Retry INSERT
ERROR: Failed to get connection from pool
Retry

View Slide

https://docs.microsoft.com/en-us/dotnet/api/system.net.httpwebrequest.timeout

View Slide


View Slide

https://dev.mysql.com/doc/connector-j/5.1/en/connector-j-reference-configuration-properties.html

View Slide

@timeout_decorator.timeout(5, timeout_exception=StopIteration)
def timed_get(url):
return requests.get(url)
https://pypi.org/project/timeout-decorator/

View Slide

Set the timeouts!

View Slide

How else could we have prevented the error?
User 1
DB
Conn
Pool
INSERT
Retry INSERT
Retry INSERT
Retry
ERROR: Failed to get connection from pool

View Slide

User 1
DB
Conn
Pool
INSERT
Timeout client side = 10s Timeout backend side = 10s
Wait 2s before Retry
INSERT
INSERT
Backing off between retries
Releasing connections
Backoff

View Slide

No jitter With jitter
https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/
Simple Exponential Backoff is not enough: Add Jitter

View Slide

Example: add jitter 0-1000ms
MAX_TRIES = 12
def get_item(self, url, n=1):
try:
res = requests.get(url)
except:
if n > MAX_TRIES:
return None
n += 1
time.sleep((2 ** n) + (random.randint(0, 1000) /
1000.0))
return self.get_item(url, n)
else:
return res

View Slide


View Slide

Idempotent operation
No additional effect if it is called more
than once with the same input parameters.

View Slide

Circuit Breaker
• Wrap a protected function
call in a circuit breaker
object, which monitors for
failures.
• If failures reach a certain
threshold, the circuit
breaker trips.
Producer Circuit Breaker Consumer
Connection
Monitoring
Timeouts
Breaking Circuit

View Slide

https://github.com/Netflix/Hystrix

View Slide

https://spring.io/guides/gs/circuit-breaker/

View Slide

Let’s talk about health checking!

View Slide

Auto Scaling group
Service A
Availability zone 1
Auto Scaling group
AWS Region
Service A
Availability zone 2
Service B
Service B
database Email
Probing for health
Cluster

View Slide

Shallow health check
Instance
Cache
node
Email
database
Cluster
Are you healthy?
yes

View Slide

Deep health check
Instance
Cache
node
Email
database
Cluster
Are you healthy?
yes
Are you healthy?
yes
yes
yes
yes

View Slide

Deep health check
Instance
Cache
node
Email
database
Cluster
Are you healthy?
no
Are you healthy?
no
yes
yes
yes

View Slide

Prioritize shallow health checks during
hard times.
Cache and be careful with logging.

View Slide

Let’s talk about load shedding.

View Slide


View Slide

Don’t be overly optimistic and take on more than you can.
Find an operational metric to reject what you cannot take in.
Favor cached and static content
Prioritize ELB health check (shallow) pings
In an overload situation you have precious resources, do not
let any of it go to waste.
Load Shedding

View Slide

Service Degradation & Fallbacks

View Slide

https://twitter.com/redditstatus/status/1116204502703493120

View Slide

Let’s talk about sharding.

View Slide


View Slide

Assign Customers to Cells
Cell 1 Cell 2 Cell 3 Cell 5 Cell 6
Cell 4

View Slide

Measure for this: blast radius

View Slide

Blast radius
• How many customers?
• What functionality?
• How many locations?

View Slide

Shuffle sharding

View Slide

Cell 4
Assign Customers to Random Cells

View Slide

Cell 4
Assign Customers to Cells

View Slide

Cell 4
Assign Customers to Random Cells

View Slide

Shuffle sharding
Nodes = 8
Shard size = 2
Combinations = 28
Overlap % customers
0 53.6%
1 42.8%
2 3.6%

View Slide

Shuffle sharding
Nodes = 100
Shard size = 5
Combinations = 75 million!
Overlap % customers
0 77%
1 21%
2 1.8%
3 0.06%
4 0.0006%
5 0.0000013%

View Slide

Shuffle sharding

View Slide

Let’s talk about chaos!

View Slide

GameDay at Amazon
Creating Resiliency Through Destruction
https://www.youtube.com/watch?v=zoz0ZjfrQ9s

View Slide

Chaos engineering
https://github.com/Netflix/SimianArmy

View Slide

“Chaos Engineering is the discipline of
experimenting on a distributed system
in order to build confidence in the system’s
capability to withstand turbulent conditions in
production.”
http://principlesofchaos.org

View Slide

Failure injection
Start small & build confidence
•Application level
•Host failure
•Resource attacks (CPU, memory, …)
•Network attacks (dependencies, latency, …)
•Region attacks

View Slide

AWS Fault Injection Simulator
Fully managed chaos engineering service on AWS
Coming soon

View Slide

Resiliency and Availability Design Patterns for the Cloud

Resiliency and Availability Design Patterns for the Cloud

Sébastien Stormacq - AWS Developer Advocate

More Decks by Sébastien Stormacq - AWS Developer Advocate

Other Decks in Programming

Featured

Transcript