Everything is Okay: Database Edition

Transcript

4. EVERYTHING IS OKAY
5. EVERYTHING IS OKAY: DATABASE EDITION
6. "A distributed system is one in which the failure of a computer you didn't even know existed can render your own computer unusable." — Leslie Lamport, 1987
7. The network is reliable Latency is zero Bandwidth is infinite The network is secure Topology doesn't change There is one administrator Transport cost is zero The network is homogeneous Everything is awesome
8. The network is reliable Latency is zero Bandwidth is infinite The network is secure Topology doesn't change There is one administrator Transport cost is zero The network is homogeneous Everything is awesome
9. Consistency: all nodes see the same data Availability: all requests get a response Partition tolerance: nodes survive arbitrary message loss
10. CP / AP / CA
11. CP / AP
12. YOU CAN'T SACRIFICE PARTITION TOLERANCE.
13. The network is reliable Latency is zero Bandwidth is infinite The network is secure Topology doesn't change There is one administrator Transport cost is zero The network is homogeneous Everything is awesome
14. The network is reliable Latency is zero Bandwidth is infinite The network is secure Topology doesn't change There is one administrator Transport cost is zero The network is homogeneous Everything is awesome
15. A typical first year for a new Google cluster: 40-80 machines with 50% packet loss dozens of DNS blips 12 router restarts 8 network maintenance events 3 router failures 1 network rewiring ...and that's just LAN incidents
16. YOU CAN'T SACRIFICE PARTITION TOLERANCE.
17. YOU CAN'T SACRIFICE PARTITION TOLERANCE.
18. YOU CAN'T SACRIFICE PARTITION
19. "Despite your best efforts, your system will experience enough faults that it will have to make a choice between reducing yield (i.e., stop answering requests) and reducing harvest (i.e., giving answers based on incomplete data). This decision should be based on business requirements."
20. CP: When faced with a network partition, stop answering requests AP: When faced with a network partition, answer requests using incomplete data
21. Make your choice, but it might still suck.
22. Consistency: really freaking hard Availability: okay I guess Partition tolerance: really freaking hard
23. PAXOS (1998)
24. PAXOS & CHUBBY (2007)
25. "There are significant gaps between the description of the Paxos algorithm and the needs of a real-world system. In order to build a real- world system, an expert needs to use numerous ideas scattered in the literature and make several relatively small protocol extensions. The cumulative effort will be substantial and the final system will be based on an unproven protocol."
26. "The fault-tolerance computing community has not developed the tools to make it easy to implement their algorithms."
27. "The fault-tolerance computing community has not paid enough attention to testing, a key ingredient for building fault-tolerant systems."
28. PAXOS & MEGASTORE (2011)
29. "While many systems use Paxos solely for locking, master election, or replication of metadata and configurations, we believe that Megastore is the largest system deployed that uses Paxos to replicate primary user data across datacenters on every write."
30. "The original Paxos algorithm is ill-suited for high-latency network links because it demands multiple rounds of communication."
31. "Megastore is perhaps the first large-scale storage system to implement Paxos-based replication across datacenters while satisfying the scalability and performance requirements of scalable web applications in the cloud."
32. RAFT (2014)
33. "In Search of an Understandable Consensus Algorithm"
34. JEPSEN (2013—)
35. PostgreSQL (CP) "...we cannot assert the state of the system for these writes."
36. PostgreSQL (CP) "...we cannot assert the state of the system for these writes."
37. Redis (CP) "Redis threw away 56% of the writes it told us succeeded."
38. MongoDB (CP and AP???) "MongoDB is neither AP nor CP. The defaults can cause significant loss of acknowledged writes. The strongest consistency offered has bugs which cause false acknowledgements..."
39. Riak (AP) "Riak’s last-write-wins is fundamentally unsafe in the presence of network partitions."
40. Zookeeper (CP) "Use Zookeeper...wherever possible [taking] advantage of tested recipes and client libraries..."
41. NuoDB (CAP counterexample???) "...most, but not all, writes made during the partition failed...and were not retried"
42. Kafka (CA???) "Kafka’s replication claimed to be CA, but in the presence of a partition, threw away an arbitrarily large volume of committed writes."
43. Cassandra (AP) "Cassandra lightweight transactions are not even close to correct."
44. RabbitMQ (CP or AP) "...in the presence of partitions, RabbitMQ clustering will not only deliver duplicate messages, but will also drop huge volumes of acknowledged messages on the floor."
45. etcd and Consul (CP) "consistent reads...allow stale reads"
46. Elasticsearch (CP) "Elasticsearch appears to lose writes...during asymmetric partitions, symmetric partitions, overlapping partitions, disjoint partitions, and even partitions which only isolate a single node once. Its convergence times are slow and the cluster can repeatably deadlock, forcing an administrator to intervene before recovery...If you are an Elasticsearch user (as I am): good luck."
47. UH...
48. Join a monastery Pay Google Hope your site never gets popular Choose CP or AP, read aphyr.com, and hope for the best
49. Join a monastery Pay Google Hope your site never gets popular Choose CP or AP, read aphyr.com, and hope for the best
50. http://aphyr.com http://codahale.com/you-cant-sacrifice-partition-tolerance http://research.google.com (Chubby, Megastore) https://ramcloud.stanford.edu/~ongaro/thesis.pdf http://www.somethingsimilar.com/2013/01/14/ notes-on-distributed-systems-for-young-bloods