Reliability of Distributed Systems

Transcript

1. Reliability of Distributed Systems - Piyush Verma
2. Every product either dies a hero or lives long enough to hit Reliability issues. 2
3. Customer Empathy No Chooran. Cost to Everything Architectures adapt to $ Priority 01 02 03 04 3 Take it and Go
4. Service receives SMS User sends SMS Remind me to buy milk at 6:30 PM to 53308 Cron gets Activated when time is right. Call the User 4 Sample Product
5. Service receives SMS User sends SMS Remind me to buy milk at 6:30 PM to 53308 Cron gets Activated when time is right. Call the User 5 Sample Product: Inbound
6. Cron gets Activated when time is right. Call the User 6 Sample Product: Outbound Service receives SMS User sends SMS Remind me to buy milk at 6:30 PM to 53308
7. 7 Inbound Connection
8. — Leslie Lamport https://www.microsoft.com/en-us/research/uploads/prod/2016 /12/Distribution.pdf “A distributed system is one in which the failure of a computer you didn’t even know existed can render your own computer unusable” 8
9. Four Flavors of Failure 9 Disk Network CPU Memory
10. Network is Reliable Intra-LAN latency is ~ Zero Network is Homogeneous Network cost is Zero 01 02 03 04 10
11. 11 Scope of Failures: Again
12. At-least one server is online All servers are below 100% All servers are responding within x ms. All of the above. 01 02 03 04 12
13. 13 #1 Server is Unavailable
14. 14 Replication Available
15. 15 Replication Available
16. 16 Available != Load Balanced
17. 17 Load Balanced
18. 18 Architecture of a Balancer
19. GCP AWS On-Prem Azure Who uses What? 19
20. Trilemma 20
21. Trilemma 21 Available Economical Endurable
22. 22 Available + Load Balanced
23. Load Balancing 23
24. 24 Monty Hall Problem: Was Marilyn vos Savant, right?
25. 25 Server-side Load Balancing Example: Fabio
26. 26 Look-aside Load Balancing Example: Consul/ DNS
27. 27 Client-side Load Balancing Example: Ribbon
28. 28 Client-side Load Balancing Example: Ribbon + Curator
29. 29 Problems
30. 30 Load Shedding
31. — Tyler McMullen https://www.infoq.com/presentations/load-balancing/ “Load Balancing is almost Impossible” 31
32. Alternate Reliability 32
33. 33 Asynchronous Architectures
34. 34 Asynchronous Architectures Example: RabbitMQ Kafka Kinesis SQS
35. Cron gets Activated when time is right. Call the User 35 Sample Product: Outbound Service receives SMS User sends SMS Remind me to buy milk at 6:30 PM to 53308 Part 1
36. 36 Outbound
37. 37 Scope of Failure: Outbound
38. 38 Retries
39. 39 Retries: Transient Failures
40. 40 Exponential Backoff: Short term Transient Failures ✋ ✋ ✋ ✋ ✋
41. 41 Circuit Breaking: Long Term Transient Failures
42. 42 Revisited
43. Dilemma 43 At-least Once Exactly-Once At-most Once
44. 44 At-most once delivery
45. 45 At-least once delivery
46. 46 Exactly once delivery
47. 47 Exactly once delivery = At-least-once Delivery + Exactly-once Processing
48. Keys to Only-Once delivery 48 Atomic Window Idempotent
49. Out-of-Order delivery 49
50. Revisit 50
51. Service receives SMS User sends SMS Remind me to buy milk at 6:30 PM to 53308 Cron gets Activated when time is right. Call the User 51 Sample Product
52. 52 Problems of State
53. 53 Problems of State
54. 54 Locked /Serialization
55. 55 Master/ Master/ Slave
56. 56 Clustering
57. Scalability 57 Data Replication Reduced Communication Logic/Data Decentralization
58. CAP Theorem [Sab topi pehna rahe] 58
59. 59 Trilemma Available Partition Consistent
60. PACeLC Theorem 60
61. Dilemma 61 Consistency Latency
62. 62 Revised Flow
63. 63 What about Spanner? What about Calvin?
64. Reliable System 64 Scalable Correct Transparent
65. Access Transparency Location Transparency Concurrency Transparency Failure Transparency 01 02 03 04 65
66. Size Scalability Geographical Scalability 01 02 66
67. Summary 67 Consistent Available Economical Low Latency
68. 68 All Put Together
69. Embrace your Bugs No Silver Bullet Cost to Everything Product First 01 02 03 04 69
70. Does anyone have any questions? piyush@piyushverma.net Thanks 70