Running in multiple data centers

The applications that we build for today's world have a lot of requirements. They need to provide the best user experience and to be always up and running. To achieve this in a massive scale you need a multi data center architecture. When we have more than one data centers, even if one of them goes down, the other can handle the traffic and your users will continue to use your application uninterrupted. Also by having datacenters in different locations around the world you can you take advantage of lower latencies and provide a better usability. But to take advantage of all those benefits you need to architect your application in a special way. During the talk we’ll explore the different multi data center configurations and the tradeoff of each one of them. We’ll also go over the ways to do failover and some useful processes to facilitate it better. Moreover, we’ll see how each layer of the application is affected by such architecture, all the way down to the database and the data model. Finally, I’ll share what technologies help Uber to run in multiple data centers and the lessons we learned by doing so.

Transcript

2. 16 – 17 November, Sofia ISTACON.ORG Running in multiple data centers By Nikolay Stoitsev
3. 16 – 17 November, Sofia ISTACON.ORG
4. 16 – 17 November, Sofia ISTACON.ORG 600+ cities
5. 16 – 17 November, Sofia ISTACON.ORG 75+ countries
6. 16 – 17 November, Sofia ISTACON.ORG 6 continents
7. 2 000 000+ drivers
9. 16 – 17 November, Sofia ISTACON.ORG How the Internet Kept Humming During 2 Hurricanes https://www.nytimes.com/2017/09/18/us/harvey-irma-internet.html
10. 16 – 17 November, Sofia ISTACON.ORG Fault tolerance
11. 16 – 17 November, Sofia ISTACON.ORG Low latency
12. 16 – 17 November, Sofia ISTACON.ORG Compliance
13. 16 – 17 November, Sofia ISTACON.ORG Data locality
14. 16 – 17 November, Sofia ISTACON.ORG Under-utilized capacity
15. 16 – 17 November, Sofia ISTACON.ORG CAP
16. 16 – 17 November, Sofia ISTACON.ORG Continuous network partition
17. 16 – 17 November, Sofia ISTACON.ORG 2 types of architecture
18. 16 – 17 November, Sofia ISTACON.ORG Active-Passive
19. DC 1 DC 2
20. DC 1 DC 2
21. 16 – 17 November, Sofia ISTACON.ORG Failover
22. 16 – 17 November, Sofia ISTACON.ORG DNS
23. 16 – 17 November, Sofia ISTACON.ORG Stateless service
24. 16 – 17 November, Sofia ISTACON.ORG Stateful service
25. DC 1 DC 2 DB 1 DB 2 Active-Passive example
26. DC 1 DC 2 DB 1 DB 2 Active-Passive example
27. DC 1 DC 2 DB 1 DB 2 Active-Passive example
28. DC 1 DC 2 Master Slave Slave Slave Real-life example
29. DC 1 DC 2 Master Slave Slave Slave HAProxy HAProxy Smart intermediary
30. DC 1 DC 2 Master Slave Slave Slave HAProxy HAProxy Smart intermediary
31. DC 1 DC 2 Slave Slave Master Slave HAProxy HAProxy Smart intermediary
32. 16 – 17 November, Sofia ISTACON.ORG All-active
33. DC 1 DC 2
34. 16 – 17 November, Sofia ISTACON.ORG Locality
35. 16 – 17 November, Sofia ISTACON.ORG Split traffic in groups
36. 16 – 17 November, Sofia ISTACON.ORG Global State
37. mod 2 DC 1 DC 2 user_id = 0 = 1 Partitioning
38. mod 3 DC 1 DC 2 user_id = 0 = 1 DC 3 = 2 Partitioning
39. 16 – 17 November, Sofia ISTACON.ORG Very inefficient
40. 16 – 17 November, Sofia ISTACON.ORG Consistent hashing DC 1 DC 3 DC 2 DC 3
41. 16 – 17 November, Sofia ISTACON.ORG Consistent hashing DC 1 DC 3 DC 2 DC 3 user_id
42. 16 – 17 November, Sofia ISTACON.ORG DNS load balancing
43. 16 – 17 November, Sofia ISTACON.ORG DC 1 DC 2 San Francisco Los Angeles New York Toronto
44. 16 – 17 November, Sofia ISTACON.ORG DC 1 DC 2 San Francisco Los Angeles New York Toronto
45. 16 – 17 November, Sofia ISTACON.ORG Database layer
46. 16 – 17 November, Sofia ISTACON.ORG No generic solution
47. 16 – 17 November, Sofia ISTACON.ORG Galera Cluster Synchronous multi-master database cluster http://galeracluster.com/
48. 16 – 17 November, Sofia ISTACON.ORG DC 1 Master Slave DC 2 Slave Master DC 3 Master Slave DC 4 Slave Master
49. 16 – 17 November, Sofia ISTACON.ORG Apache Cassandra http://cassandra.apache.org/
50. 16 – 17 November, Sofia ISTACON.ORG Linear scalability Fault-tolerance Commodity hardware
51. 16 – 17 November, Sofia ISTACON.ORG Designed for multiple data centers
52. 16 – 17 November, Sofia ISTACON.ORG Apache Mesos http://mesos.apache.org/
53. 16 – 17 November, Sofia ISTACON.ORG Application Layer
54. 16 – 17 November, Sofia ISTACON.ORG Apache Kafka
55. 16 – 17 November, Sofia ISTACON.ORG uReplicator https://github.com/uber/uReplicator
56. 16 – 17 November, Sofia ISTACON.ORG https://eng.uber.com/ureplicator/
57. 16 – 17 November, Sofia ISTACON.ORG https://eng.uber.com/ureplicator/
58. 16 – 17 November, Sofia ISTACON.ORG Cherami https://github.com/uber/cherami-server
59. 16 – 17 November, Sofia ISTACON.ORG Multi-zone topics Producer Producer Topic Topic Consumer Group Consumer Group replication
60. 16 – 17 November, Sofia ISTACON.ORG Multi-zone consumers Producer Topic Topic Consumer Group Consumer Group replication offset sync
61. 16 – 17 November, Sofia ISTACON.ORG https://eng.uber.com/cherami/
62. 16 – 17 November, Sofia ISTACON.ORG Lessons learned
63. 16 – 17 November, Sofia ISTACON.ORG Total dev time Time thinking about failover
64. 16 – 17 November, Sofia ISTACON.ORG Total dev time Time thinking about failover
65. 16 – 17 November, Sofia ISTACON.ORG Failover testing
66. 16 – 17 November, Sofia ISTACON.ORG Failure testing
67. 16 – 17 November, Sofia ISTACON.ORG Super smart clients
68. 16 – 17 November, Sofia ISTACON.ORG “ The best way to avoid failure is to fail constantly. http://techblog.netflix.com/2010/12/5-lessons-weve-learned-using-aws.html
69. 16 – 17 November, Sofia ISTACON.ORG Thank you! @stoitsev Nikolay Stoitsev http://careersinfo.uber.com/sofia-engineering