CAP Theorem: You don’t need CP, you don’t want AP, and you can’t have CA

Video: https://www.youtube.com/watch?v=hUd_9FENShA

CAP Theorem is everywhere: “Consistency, Availability, Partition tolerance — choose any two!” But it is oversimplified and misunderstood more often than not. CAP’s consistency isn’t what most people think it is; CAP’s availability isn’t what most people think it is; what does partition-tolerance even mean?

In this talk we’ll explore the CAP-theorem and understand what it is really asserting. We’ll understand that just calling a system out as CP or AP (or even CA) is pretty pointless, and learn to judge them beyond the simple monikers. We’ll also analyse some popular databases of the world (MySQL, ZooKeeper, Cassandra, Kafka etc.) with this framework.

Transcript

1. 16th July 2015 Siddhartha Reddy Partition-tolerance Consistency Availability You don’t need CP You don’t want AP You can’t have CA
2. Show of hands https://www.flickr.com/photos/dotbenjamin/2765083201/
3. Show of hands ✤ Heard of CAP Theorem? https://www.flickr.com/photos/dotbenjamin/2765083201/
4. Show of hands ✤ Heard of CAP Theorem? ✤ Used CAP Theorem? https://www.flickr.com/photos/dotbenjamin/2765083201/
5. Show of hands ✤ Heard of CAP Theorem? ✤ Used CAP Theorem? ✤ Read the paper/proof? https://www.flickr.com/photos/dotbenjamin/2765083201/
6. History
7. History ✤ Proposed by Dr. Eric Brewer
 [Symposium on Principles of Distributed Computing, 2000]
8. History ✤ Proposed by Dr. Eric Brewer
 [Symposium on Principles of Distributed Computing, 2000] ✤ Proved by Gilbert & Lynch
 [Brewer’s Conjecture and the Feasibility of Consistent, Available, Partition-Tolerant Web Services, 2002] ✤ “a shared-data system can have at most two of the three following properties: Consistency, Availability, and tolerance to network Partitions”
19. Partition! Choose
 Availability Choose
 Consistency
20. CAP Theorem ✤ Consistency, Availability, Partition-tolerance
 — “pick any two”
21. CAP Theorem ✤ Consistency, Availability, Partition-tolerance
 — “pick any two” ✤ CP: Consistent & Partition-tolerant
 (sacrifice Availability)
22. CAP Theorem ✤ Consistency, Availability, Partition-tolerance
 — “pick any two” ✤ CP: Consistent & Partition-tolerant
 (sacrifice Availability) ✤ AP: Available & Partition-tolerant
 (sacrifice Consistency)
23. CAP Theorem ✤ Consistency, Availability, Partition-tolerance
 — “pick any two” ✤ CP: Consistent & Partition-tolerant
 (sacrifice Availability) ✤ AP: Available & Partition-tolerant
 (sacrifice Consistency) ✤ CA: Consistent & Available
 (sacrifice Partition-tolerance)
24. Consistent & Available — An existential crisis Or why you can’t have CA
25. CAP-Partitions “When a network is partitioned, all messages sent from nodes in one component of the partition to nodes in another component are lost” https://aphyr.com/posts/285-call-me-maybe-riak
26. CA: Sacrificing Partition-tolerance
27. CA: Sacrificing Partition-tolerance Partitions can’t or won’t happen
28. CA: Sacrificing Partition-tolerance Partitions can’t or won’t happen A single-node database → nothing to partition. Yay!
29. CA: Sacrificing Partition-tolerance Partitions can’t or won’t happen A single-node database → nothing to partition. Yay! “The network is reliable” ✤ LAN, high quality equipment etc.
30. Reality-check: Networks Fail Image Credits: Switch, PDU, Wiring, Congested Network
31. Partitions are a reality Network equipment failure Power/cooling failure Network congestion
32. Partitions are a reality Network equipment failure Power/cooling failure Network congestion Garbage Collection
33. Partitions are a reality Network equipment failure Power/cooling failure Network congestion Garbage Collection Misconfigurations
34. Partitions are a reality Network equipment failure Power/cooling failure Network congestion Garbage Collection Misconfigurations Software bugs
35. You can’t wish away Partitions! You can’t have CA
36. Consistency — A costly fare Or why you don’t need CP
37. CAP-Consistency ✤ Linearizable consistency ✤ “requests of the distributed shared memory to act as if they were executing on a single node, responding to operations one at a time”
38. CAP-Consistency ✤ Linearizable consistency ✤ “requests of the distributed shared memory to act as if they were executing on a single node, responding to operations one at a time” Once an operation is complete, it will be visible to all.
39. Linearizability is hard
40. Linearizability is hard https://aphyr.com/posts/313-strong-consistency-models
41. Linearizability is hard ✤ Two-phase commits https://aphyr.com/posts/313-strong-consistency-models
42. Linearizability is hard ✤ Two-phase commits ✤ Sync replication https://aphyr.com/posts/313-strong-consistency-models
43. Linearizability is hard ✤ Two-phase commits ✤ Sync replication ✤ Memory access in CPU — not linearizable
 (memory barriers can be used to make it linearizable) https://aphyr.com/posts/313-strong-consistency-models
44. Linearizability is hard ✤ Two-phase commits ✤ Sync replication ✤ Memory access in CPU — not linearizable
 (memory barriers can be used to make it linearizable) ✤ Variables — not linearizable
 (volatile keyword in Java makes them linearizable) https://aphyr.com/posts/313-strong-consistency-models
45. Linearizability is hard ✤ Two-phase commits ✤ Sync replication ✤ Memory access in CPU — not linearizable
 (memory barriers can be used to make it linearizable) ✤ Variables — not linearizable
 (volatile keyword in Java makes them linearizable) ✤ Relational DBs https://aphyr.com/posts/313-strong-consistency-models
46. Linearizability — extreme form of consistency
 & extremely costly to provide; 
 possible to build useful systems without it You don’t need CP
47. Availability: The blue pill Or why you don’t want AP https://www.flickr.com/photos/mlemos/3911634112
48. CAP-Availability ✤ “every request received by a non- failing node in the system must result in a response”
49. High Availability (HA)
50. High Availability (HA) ✤ Uptime: 99%, 99.9%, 99.99%, 99.999%
51. High Availability (HA) ✤ Uptime: 99%, 99.9%, 99.99%, 99.999% ✤ CAP-availability has little to do with
 High Availability
52. High Availability (HA) ✤ Uptime: 99%, 99.9%, 99.99%, 99.999% ✤ CAP-availability has little to do with
 High Availability CAP Availability High Availability non-failing nodes the whole system unbounded response time realistic
 response times
53. High Availability (HA) ✤ Uptime: 99%, 99.9%, 99.99%, 99.999% ✤ CAP-availability has little to do with
 High Availability ✤ All nodes returning empty response
 ⇒ CAP-available! CAP Availability High Availability non-failing nodes the whole system unbounded response time realistic
 response times
54. High Availability (HA) ✤ Uptime: 99%, 99.9%, 99.99%, 99.999% ✤ CAP-availability has little to do with
 High Availability ✤ All nodes returning empty response
 ⇒ CAP-available! ✤ No response from any node
 ⇒ CAP-available! CAP Availability High Availability non-failing nodes the whole system unbounded response time realistic
 response times
55. Your goal is High Availability;
 CAP-availability has little overlap with that You don’t want AP
56. Consistency & Availability Relationship status: It’s complicated
57. Consistency vs. Availability ✤ Trade-offs between consistency & availability are real
58. Consistency vs. Availability ✤ Trade-offs between consistency & availability are real ✤ AP to CP is a spectrum
 and “the whole space is useful” – Eric Brewer
 AP CP Availability Consistency
59. Consistency vs. Availability ✤ Trade-offs between consistency & availability are real ✤ AP to CP is a spectrum
 and “the whole space is useful” – Eric Brewer
 ✤ Most systems are neither CP nor AP, don’t need CP or AP ✤ eventual consistency & high availability is usually good enough AP CP Availability Consistency
60. Consistency, Availability & more ✤ Trade-offs between consistency & availability are real;
 there are also trade-offs with latency, operational simplicity etc. AP CP Availability Consistency Latency Simplicity
61. Consistency, Availability & more ✤ Trade-offs between consistency & availability are real;
 there are also trade-offs with latency, operational simplicity etc. AP CP Availability Consistency Latency Simplicity
62. Consistency, Availability & more ✤ Trade-offs between consistency & availability are real;
 there are also trade-offs with latency, operational simplicity etc. ✤ CAP-Theorem is an incomplete tool
 for analysing the these trade-offs AP CP Availability Consistency Latency Simplicity
63. PACELC: A CAP-theorem alternative
64. PACELC: A CAP-theorem alternative AP CP Availability Consistency Latency Consistency
65. PACELC (pass-elk)
66. PACELC (pass-elk) If there is a Partition, how does the system trade-off Availability & Consistency
67. PACELC (pass-elk) If there is a Partition, how does the system trade-off Availability & Consistency Else, how does the system trade-off Latency & Consistency
68. AP CP Availability Consistency Latency Consistency
69. AP CP Availability Consistency Latency Consistency PC/EC
70. AP CP Availability Consistency Latency Consistency PA/EL PC/EC
71. AP CP Availability Consistency Latency Consistency PA/EL PA/EC PC/EC
72. AP CP Availability Consistency Latency Consistency PC/EL PA/EL PA/EC PC/EC
73. Applying the theory Examining some real-world databases
75. AP CP Availability Consistency Latency Consistency
76. AP CP Availability Consistency Latency Consistency Async Replication
77. AP CP Availability Consistency Latency Consistency Semi-sync Replication Async Replication
79. AP CP Availability Consistency Latency Consistency
80. AP CP Availability Consistency Latency Consistency read
81. AP CP Availability Consistency Latency Consistency read sync+read
83. AP CP Availability Consistency Latency Consistency
84. AP CP Availability Consistency Latency Consistency Write-consistency: ANY
 Hinted-handoffs: on
 Read-consistency: ONE
85. AP CP Availability Consistency Latency Consistency Write-consistency: ANY
 Hinted-handoffs: on
 Read-consistency: ONE Write-consistency: ALL
 Read-consistency: ALL
87. AP CP Availability Consistency Latency Consistency
88. AP CP Availability Consistency Latency Consistency required.acks=0
89. AP CP Availability Consistency Latency Consistency required.acks=0 required.acks=-1
 min.isr=<#replicas>
90. AP CP Availability Consistency Latency Consistency required.acks=0 required.acks=-1
 min.isr=1 required.acks=-1
 min.isr=<#replicas>
91. AP CP Availability Consistency Latency Consistency
92. AP CP Availability Consistency Latency Consistency PNUTS
93. Concluding
94. Concluding ✤ Availability vs. Consistency vs. Latency ✤ Trade-offs are real
95. Concluding ✤ Availability vs. Consistency vs. Latency ✤ Trade-offs are real ✤ A business (not engineering) decision
96. Concluding ✤ Availability vs. Consistency vs. Latency ✤ Trade-offs are real ✤ A business (not engineering) decision ✤ “Would you rather be down or show wrong prices?” ✤ “Would you rather be slow or show wrong prices?”
97. Concluding ✤ Availability vs. Consistency vs. Latency ✤ Trade-offs are real ✤ A business (not engineering) decision ✤ “Would you rather be down or show wrong prices?” ✤ “Would you rather be slow or show wrong prices?” ✤ Financial transactions?
98. Concluding ✤ Availability vs. Consistency vs. Latency ✤ Trade-offs are real ✤ A business (not engineering) decision ✤ “Would you rather be down or show wrong prices?” ✤ “Would you rather be slow or show wrong prices?” ✤ Financial transactions? ✤ Finance industry runs on availability!
101. Thank You. Siddhartha Reddy @sids sid@flipkart.com References: https://pinboard.in/u:sids/t:fifthel2015/