Resilient_Cloud_Architecture_Design_Patterns_.pdf

Resilient architecture is crucial for all cloud implementations. In this talk, we explore different design patterns to make a distributed application more resilient.

As part of this journey, for any process, we need to ask what if something goes wrong? Then, plan a course of action to the process auto heal without any human intervention and how to lower risks by performing canary deployments. Design starts with at first understanding of requirements and performing empathy map and value chain analysis.

Thinking application as stateless for all the API calls makes the system available most of the time requires creating a cache for common distributed data. Next, we examine how to deal with cascading failures, and timeouts scenarios. Applications, as part of auto-healing, need to Detect, Prevent, Recover, Mitigate, Complement so that the service is resilient.

Key takeaways for the audience are as follows:

Resiliency is essential for any feature in cloud
Understanding the value chain is critical to identify failure points
Challenges come in determining if there is a failure and design the system for auto-healing.
The focus should be first to prevent a failure to occur.
Identifying key challenges in your company and tools and techniques to auto-heal and provide a sustainable solution

Transcript

1. Resilient Cloud Architecture Design Patterns Rohit Bhardwaj Twitter: rbhardwaj1 Chief Architect
2. Fail points Resiliency Cell Based Auto Scaling Reactive Timeouts Eventual consistency Caching Failure Injection
3. Fail points
7. https://eng.uber.com/wp-content/uploads/2015/01/SOA_overview.png
8. EMPATHY MAP
9. https://app.creately.com/diagram/vector/i6pzy90c2
10. Initial über
11. Availability
13. Availability • Elimination of single points of failure. • Reliable crossover • Detection of failures as they occur.
15. Operational cost per trip
16. Response time
17. https://creately.com/jupiter/diagram/image/i9rx49yi1
20. Fail points Resiliency Cell Based Auto Scaling Reactive Timeouts Eventual consistency Caching Failure Injection
21. Resiliency
25. When something is strong and able to recover from damage quickly, call it resilient. https://www.vocabulary.com/dictionary/resilient
26. Availability is the assurance that an enterprise’s IT infrastructure has suitable recoverability and protection from system failures, natural disasters or malicious attacks. https://www.gartner.com/it-glossary/availability
28. Recovery Point Objective RPO Recovery Time Objectivre RTO WRT: Work recovery time MTDT: Maximum Tolerable Down Time = RTO + WRT
29. Serial Availability Service A Service B Service Availability Downtime Service A 99% 3.65 days/year Service B 99.99% 52 minutes/year A + B 98.99% 3.69 days/year A = A a A b
30. Parallel Availability Service A Service B Service Availability Downtime Service A 99% 3.65 days/year Service B 99.99% 52 minutes/year A + B 99.9999% 31 seconds year A = 1-(1-A a )2
35. Zonal services Regional services EC2 instances S3, EFS, Dynamo DB EBS volumes Fargate, Kinesis EMR clusters API Gateway, Lambda NAT gateways SQS
37. Availability Zone failure Zone A Zone B Zone C Zone A EC2 Zone B EC2 Zone C EC2 AWS SQS Service
38. Zone A Zone B Zone C Zone A EC2 Zone B EC2 Zone C EC2 AWS SQS Service Availability Zone failure
39. Fail points Resiliency Cell Based Auto Scaling Reactive Timeouts Eventual consistency Caching Failure Injection
40. Cell Based Architecture
41. Cell-based architecture Application Load Balancer Compute Storage Application Load Balancer Compute Storage Application Load Balancer Compute Storage Cell 0 Cell 1 Cell 2 Cell Router Regional Service
42. Region service Zone A Zone B Zone C Zone A Zone B Zone C AWS SQS Service AWS SQS Service Cell AWS SQS Service Cell AWS SQS Service Cell
43. Region service outage Zone A Zone B Zone C Zone A Zone B Zone C AWS SQS Service AWS SQS Service Cell AWS SQS Service Cell AWS SQS Service Cell
44. Zonal service Zone A Zone B Zone C EC2 Service Cell EC2 Service Cell EC2 Service Cell EC2 Service Cell EC2 Service Cell EC2 Service Cell EC2 Service Cell EC2 Service Cell EC2 Service Cell Zone A Zone B Zone C EC2 Service EC2 Service EC2 Service
45. Zonal service outage Zone A Zone B Zone C EC2 Service Cell EC2 Service Cell EC2 Service Cell EC2 Service Cell EC2 Service Cell EC2 Service Cell EC2 Service Cell EC2 Service Cell EC2 Service Cell Zone A Zone B Zone C EC2 Service EC2 Service EC2 Service
46. Partial Zonal service outage Zone A Zone B Zone C EC2 Service Cell EC2 Service Cell EC2 Service Cell EC2 Service Cell EC2 Service Cell EC2 Service Cell EC2 Service Cell EC2 Service Cell EC2 Service Cell Zone A Zone B Zone C EC2 Service EC2 Service EC2 Service
47. Cell Based Architecture • Workload isolation • Failure containment • Testability • Manageability
48. Cell-based architecture Application Load Balancer Compute Storage Application Load Balancer Compute Storage Application Load Balancer Compute Storage Cell 0 Cell 1 Cell 2 Cell Router Regional Service
49. Scale-out Scale-up Master Worker Worker T3 EC2 M4 EC2 XL EC2
50. Compute Compute Compute Service A Cell Router all Customers Regional Service Service B Service C
51. Compute Compute Compute Service A Cell Router all Customers Regional Service Service B Service C
52. Compute Compute Compute Service A Cell Router all Customers Regional Service Service B Service C
53. Compute Compute Compute Service A Cell Router by Customers Regional Service Service B Service C Blast radius = All customers
55. Application Load Balancer Compute Storage Cell 0 Cell Router by Customer Regional Service A B Application Load Balancer Compute Storage Cell 1 Cell Router by Customer C D Application Load Balancer Compute Storage Cell 2 Cell Router by Customer E F Application Load Balancer Compute Storage Cell 3 Cell Router by Customer G H
56. Application Load Balancer Compute Storage Cell 0 Cell Router by Customer Regional Service A B Application Load Balancer Compute Storage Cell 1 Cell Router by Customer C D Application Load Balancer Compute Storage Cell 2 Cell Router by Customer E F Application Load Balancer Compute Storage Cell 3 Cell Router by Customer G H
57. Application Load Balancer Compute Storage Cell 0 Cell Router by Customer Regional Service A B Application Load Balancer Compute Storage Cell 1 Cell Router by Customer C D Application Load Balancer Compute Storage Cell 2 Cell Router by Customer E F Application Load Balancer Compute Storage Cell 3 Cell Router by Customer G H A C B A C E B D
58. Application Load Balancer Compute Storage Cell 0 Cell Router by Customer A B Application Load Balancer Compute Storage Cell 1 Cell Router by Customer C D Application Load Balancer Compute Storage Cell 2 Cell Router by Customer E F Application Load Balancer Compute Storage Cell 3 Cell Router by Customer G H A C B A C E B D Blast radius = Customers / Combinations
59. Shuffle Sharding Nodes = 7 Shard size = 2 Combinations = 8! / (2! * (8-2)! = 28 Overlap % customers impacted 0 53.6% 1 42.8% 2 3.6% Combinations (n/k) = n! / (k! (n-k)! https://threadreaderapp.com/thread/1034492056968736768.html
60. Shuffle Sharding Nodes = 100 Shard size = 5 Overlap % customers impacted 0 77% 1 21% 2 1.8% 3 0.06% 4 0.0006% 5 0.0000013% https://threadreaderapp.com/thread/1034492056968736768.html
61. Fail points Resiliency Cell Based Auto Scaling Reactive Timeouts Eventual consistency Caching Failure Injection
62. Auto Scaling
63. Autoscaling
64. https://aws.amazon.com/blogs/aws/new-predictive-scaling-for-ec2-powered-by-machine-learning/
69. CloudWatch
70. Fail points Resiliency Cell Based Auto Scaling Reactive Timeouts Eventual consistency Caching Failure Injection
71. https://www.reactivemanifesto.org/
74. Sync vs Async
75. Amplifyjs
76. Issue: Infrastructure failures causing delays • Network congestion or network partition • Request contention in non-scalable infrastructure services • I/O bandwidth saturation in storage systems • Software failures in shared services • Denial-of-service attacks that trigger service failure or resource exhaustion
77. Issue: Latency between New York and San Francisco • 13.8 ms • 27.6 ms (Round trip) • As per AT&T: 70 ms Distance, as the crow flies, between New York and San Francisco = 2,565 miles; speed of light = 186,000 miles/second.
78. Service A depends on B which depends on C http://www.stratoscale.com/wp-content/uploads/imageblog14.2.png
79. Solution 1: Co-locate http://associationnow.wpengine.netdna-cdn.com/wp-content/uploads/2012/09/0928_team-800x480.jpg
80. Solution 2: Reducing the number of network requests • REST API design • Ask for Zip and phone in same REST request • Consolidate small details into one form
81. Bandwidth Aware
82. Reduce chattiness http://techblog.netflix.com/2013/01/optimizing-netflix-api.html
83. Fail points Resiliency Cell Based Auto Scaling Reactive Timeouts Eventual consistency Caching Failure Injection
84. Timeouts and retries
85. Web App DB connect ion Pool DB Frontend timeout Backend timeout Database timeout INSERT INSERT INSERT INSERT
86. Web App DB connect ion Pool DB Frontend timeout Backend timeout Database timeout INSERT INSERT INSERT INSERT RETRY RETRY
88. Python - infinity timeout! • connect_timeout=31 # number of seconds to wait for connection to be established • read_timeout=5 # maximum number of seconds between data reads • r = requests.get('https://github.com', timeout=(connect_timeout, read_timeout))
90. MYSQL 5.7 or higher • SELECT /*+ MAX_EXECUTION_TIME(1000) */ status, count(*) FROM articles GROUP BY status ORDER BY status; • SET SESSION MAX_EXECUTION_TIME=2000; • SET GLOBAL MAX_EXECUTION_TIME=2000;
91. “INFINITY” IS A BAD DEFAULT TIMEOUT
92. Web App DB connect ion Pool DB Frontend timeout Backend timeout Database timeout INSERT INSERT INSERT INSERT
93. Web App DB connect ion Pool DB Frontend timeout = 10s Backend timeout 10s Database timeout 10s INSERT INSERT INSERT INSERT Wait 2s before retry Wait 4s before retry Wait 8s before retry Backoff between retries Release connections
94. https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/
95. Netflix case study
99. Cached trending movies Service degradation
100. Component microservices
101. Prioritization traffic - SLAs
102. Circuit Breaker
103. • A request to a remote service timed out. • The request queue for the remote service is full, indicating that the remote service is unable to handle additional requests.
104. Fallback Strategy • Fail transparently. Generates a custom response • Fail silently. Returns a null value for the request. • Fail fast. Generates a 5xx response code
105. Circuit Breaker demo https://github.com/mstine/2016_CloudNativeAppArchWorkshop
106. https://github.com/Netflix/Hystrix/tree/master/hystrix-dashboard
107. Fail points Resiliency Cell Based Auto Scaling Reactive Timeouts Eventual consistency Caching Failure Injection
109. Web App DB connect ion Pool DB Frontend timeout Backend timeout Database timeout INSERT INSERT INSERT INSERT
110. Web App DB connect ion Pool DB Frontend timeout Backend timeout Database timeout INSERT INSERT INSERT INSERT Wait 2s before retry Insert already in database!
111. big data http://mattturck.com/wp-content/uploads/2016/03/Big-Data-Landscape-2016-v18-FINAL.png
112. Web App DB connect ion Pool DB Frontend timeout Backend timeout Database timeout INSERT Wait 2s before retry HOW TO DO UPSERT? UPSERT UPSERT UPSERT
113. Shallow health check Shallow: Are you healthy? OK Health Check
114. Deep health check Deep: Are you healthy? Failed Health Check
116. Database resilience
117. db sharding
119. order by CREATE TABLE crossfit_gyms_by_location ( country_code text, state_province text, city text, gym_name text, PRIMARY KEY (country_code, state_province, city, gym_name) ) WITH CLUSTERING ORDER BY (state_province DESC, city ASC, gym_name ASC);
120. hashed values • country_code | state_province | city | gym_name • --------------+----------------+---------------+-------------------------- • CAN | ON | Toronto | CrossFit Leslieville • CAN | ON | Toronto | CrossFit Toronto • CAN | BC | Vancouver | CrossFit BC • CAN | BC | Vancouver | CrossFit Vancouver • USA | NY | New York | CrossFit Metropolis • USA | NY | New York | CrossFit NYC • USA | NV | Las Vegas | CrossFit Las Vegas • USA | NV | Las Vegas | Kaizen CrossFit • USA | CA | San Francisco | LaLanne Fitness CrossFit • USA | CA | San Francisco | San Francisco CrossFit
121. Transition state do not store in database
124. Big data characteristics https://imasaikirangeek.files.wordpress.com/2014/05/defining-big-data1.png
127. https://www.qubole.com/resources/solution/best-use-cases-for-big-data-analytics/
129. inserting/UPDATING data • INSERT INTO users (email, bio, birthday, active) VALUES (‘john.crost@cnn.com’, ‘Coach’, 646464676600, true); • UPDATE users SET email=‘john.crost@cnn.com’, bio=‘Coach’, birthday=646464676600, active=true)
130. upsert • Under the hood, INSERT and UPDATE are same • INSERT = UPDATE • BOTH OPERATIONS REQURE PRIMARY KEY
131. deleting data • Deleting a row • DELETE FROM customers WHERE id = '2829'; • Deleting a column • DELETE email FROM customers
 WHERE id = '2829'; • UPDATE customers SET email = null
 WHERE id = '2829'; • INSERT INTO customers (id, email)
 VALUES ('2829', null);
132. Idempotent HTTP method IDEMPOTENT SAFETY GET Yes Yes HEAD Yes Yes PUT Yes No DELETE Yes No POST No No PATCH No No
133. Communication
136. How to get Latest Customer Orders http://microservices.io/patterns/data/database-per-service.html
137. Get Latest Customer Orders • API Composition: Join in API side • Get Customers • Get Most recent Orders
138. Command Query Responsibility Segregation (CQRS) https://blogs.vmware.com/management/2013/02/scaling-in-complex-domains-using-cqrs-axon-and-spring-insight.html
139. Find Latest orders of Customer
140. How to update Customer data?
144. What about cancellation? https://msdn.microsoft.com/en-us/library/dn589804.aspx
146. http://microservices.io/patterns/data/saga.html
147. Fail points Resiliency Cell Based Auto Scaling Reactive Timeouts Eventual consistency Caching Failure Injection
148. Caching
149. 2007, Greg Linden Amazon 100ms delay cost === 1 % sales
150. For Impatient Web Users, an Eye Blink Is Just Too Long to Wait https://www.nytimes.com/2012/03/01/technology/impatient-web-users-flee-slow-loading-sites.html?pagewanted=all&_r=0 250 ms is magic number
151. images, videos, CSS, JavaScript, and fonts in-memory caching page caching Redis or Memcached
152. SYN Flood
153. DDoS attacks
154. cache expiration policy cache eviction memory size soft-TTL: 30 minutes hard-TTL: If refresh fails
155. 10000 requests Cache miss request coalescing “waiting rooms” 
156. Cache-aside See Cache then DB Inline-cache
157. Fail points Resiliency Cell Based Auto Scaling Reactive Timeouts Eventual consistency Caching Failure Injection
159. Principles of Chaos Engineering https://principlesofchaos.org/
160. Fallacies of Distributed Systems • The network is reliable • Latency is zero • Bandwidth is infinite • The network is secure • Topology doesn't change • There is one administrator • Transport cost is zero • The network is homogeneous
162. Chaos Gorilla
163. Cassandra maintenance
164. Isolated Regions
165. Failure Modes and Effects
168. AWS Security Epics
171. Fail points Resiliency Cell Based Auto Scaling Reactive Timeouts Eventual consistency Caching Failure Injection